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OVERVIEW:  THE  DESIGN,  IMPLEMENTATION,  AND  ANALYSIS 
OF  THE  1996  STATE  ASSESSMENT  PROGRAM 
IN  MATHEMATICS1 

Nancy  L Allen  and  John  Mazzeo 
Educational  Testing  Service 

1.1  OVERVIEW 

In  April  1988,  Congress  reauthorized  the  National  Assessment  of  Educational  Progress 
(NAEP)  and  added  a new  dimension  to  the  program — voluntary  state-by-state  assessments  on  a 
trial  basis  in  1990  and  1992,  in  addition  to  continuing  the  national  assessments  that  NAEP  had 
conducted  since  its  inception.  In  1994,  Congress  authorized  a third  Trial  State  Assessment  for 
administration  in  1994.  It  should  be  noted  that  the  word  trial  in  Trial  State  Assessment  refers  to 
the  Congressionally  mandated  trial  to  determine  whether  such  assessments  can  yield  valid, 
reliable  state  representative  data.  Enough  experience  had  been  gained  for  Congress  to  authorize 
State  Assessments,  rather  than  Trial  State  Assessments,  to  be  conducted  in  1996.  In  this  report, 
we  will  refer  to  the  voluntary'  state-by-state  assessment  program  as  the  State  Assessment 
program.  The  State  Assessment  program,  which  is  designed  to  provide  representative  data  on 
achievement  for  participating  jurisdictions,  is  distinct  from  the  assessment  designed  to  provide 
nationally  representative  data,  referred  to  in  this  report  as  the  national  assessment.  (This 
terminology  is  also  used  in  all  other  reports  of  the  1996  assessment  results.)  All  instruments  and 
procedures  used  in  the  1990,  1992,  1994,  and  1996  state  and  national  assessments  were 
previously  piloted  in  field  tests  conducted  in  the  year  prior  to  each  assessment. 

The  1990  Trial  State  Assessment  program  collected  information  on  the  mathematics 
knowledge,  skills,  understanding,  and  perceptions  of  a representative  sample  of  eighth-grade 
students  in  public  schools  in  37  states,  the  District  of  Columbia,  and  two  territories.  The  second 
phase  of  the  Trial  State  Assessment  program,  conducted  in  1992,  collected  information  on  the 
mathematics  knowledge,  skills,  understanding,  and  perceptions  of  a representative  sample  of 
fourth-  and  eighth-grade  students  and  the  reading  skills  and  understanding  of  a representative 
sample  of  fourth-grade  students  in  public  schools  in  41  states,  the  District  of  Columbia,  and  two 
territories. 

The  1994  Trial  State  Assessment  program  once  again  assessed  the  reading  skills  and 
understanding  of  representative  samples  of  fourth-grade  students,  this  time  in  44  participating 
jurisdictions.  The  1994  program  broke  new  ground  in  two  ways.  The  1994  NAEP  authorization 
called  for  the  assessment  of  samples  of  both  public-  and  nonpublic-school  students.  Thus,  for  the 
first  time  in  NAEP,  jurisdiction-level  samples  of  students  from  Catholic  schools,  other  religious 
schools  and  private  schools,  Department  of  Defense  Domestic  Dependent  Elementary  and 
Secondary  Schools  (DDESS),  and  Bureau  of  Indian  Affairs  (BIA)  schools  were  added  to  the 
Trial  Stale  Assessment  program.  Second,  samples  of  students  from  the  Department  of  Defense 


1 Nancy  L.  Allen  is  the  Director  of  Data  Analysis  and  Scaling.  NAEP  Research.  Educational  resting  Service  John 
Ma//co  is  the  Director  of  NAEP  Reporting.  Educational  Testing  Service. 
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Dependents  Schools  (DoDDS)  schools  participated  as  a jurisdiction,  along  with  the  states  and 
territories  that  have  traditionally  had  the  opportunity  to  participate  in  the  Trial  State  Assessment 
program. 

The  1996  State  Assessment  program,  described  in  this  report,  again  collected 
information  on  the  mathematics  knowledge,  skills,  understanding,  and  perceptions  of  a 
representative  sample  of  fourth-  and  eighth-grade  students  for  a third  time.  In  addition,  grade  8 
public-  and  nonpublic-school  students  were  assessed  in  science  (see  the  Technical  Report  of  the 
NAEP  1996  State  Assessment  Program  in  Science , Allen,  Swinton,  & Zelenak,  1996). 

A special  feature  of  the  1996  State  Assessments  was  the  introduction  of  new  rules  for 
student  inclusion  in  NAEP  assessments.  In  order  to  assure  that  the  mathematics  results  for  state 
assessments  in  1990,  1992,  and  1996  are  comparable,  half  of  the  schools  selected  for 
participation  in  the  1 996  assessment  used  the  old  inclusion  rules  to  determine  whether  students 
should  be  included  in  the  assessment  and  the  other  half  used  the  new  inclusion  rules.  In  addition 
to  the  two  groups  of  schools  using  the  old  and  new  inclusion  rules  without  offering  students 
special  testing  accommodations,  the  1996  national  assessment  included  a third  group  of  schools 
that  used  the  new  inclusion  rules  and  offered  students  within  those  schools  the  accommodations 
to  the  standard  NAEP  administration  procedures.  More  details  on  the  procedures  for  student 
exclusion  are  presented  in  the  report  on  field  procedures  for  the  1996  State  Assessment  program 
(Westat,  Inc.,  1996).  More  details  on  the  procedures  used  for  student  exclusion  are  presented  in 
the  report  on  field  procedures  for  the  1996  State  Assessment  program  (Westat,  1996). 

The  accommodations  provided  by  NAEP  in  the  national  assessments  were  meant  to 
match  those  specified  in  the  student’s  individualized  education  plan  (1EP)  or  those  ordinarily 
provided  in  the  classroom  for  testing  situations.  The  most  common  accommodation  was  extended 
time.  In  the  State  Assessment,  no  special  accommodations  were  offered. 

The  old  and  new  inclusion  rules  are  applied  only  when  a student  has  been  categorized  in 
his  or  her  IEP  as  a student  with  disabilities  (SD)  or  as  a student  with  limited  English  proficiency 
(LEP);  all  other  students  are  asked  to  participate  in  the  assessment.  For  this  reaeon,  the  sample  of 
students  that  were  selected  for  most  analysis  and  reporting  purposes  consisted  of  students  from 
schools  using  either  set  of  inclusion  rules  that  were  not  categorized  as  SD  or  LEP  students  and 
students  from  the  schools  using  the  old  inclusion  rules  that  were  categorized  as  SD  or  LEP.  The 
advantage  of  this  reporting  sample  is  that  it  preserves  trend  with  previous  assessments  and  it 
makes  use  of  most  of  the  data  from  the  assessment. 

Special  analyses  that  used  the  national  mathematics  assessment  data  to  compare  the  old 
and  new  inclusion  rules  and  examine  the  effect  of  offering  testing  accommodations,  indicated 
little  difference  in  proportions  of  students  included  in  the  assessment  using  the  old  and  new 
inclusions.  More  students  were  included  in  the  assessment  when  they  were  offered 
accommodations;  however,  a portion  of  students  who  would  have  participated  in  the  assessment 
under  standard  conditions  were  assessed  with  accommodations  when  they  were  offered.  A result 
of  this  is  that  fewer  students  were  assessed  under  standard  conditions  when  accommodations 
were  offered. 

Table  1-1  lists  the  jurisdictions  that  participated  in  the  1996  State  Assessment  program. 
Over  125,000  students  at  each  grade  participated  in  the  1996  State  Assessments  in  the 
jurisdictions  shown.  Students  were  administered  the  same  assessment  booklets  that  were  used  in 
either  NAEP’s  1996  national  mathematics  or  national  science  assessments. 


Table  1-1 

Jurisdictions  Participating  in  the  1996  State  Assessment  Program  in  Mathematics 


Jurisdictions 


Alabama 

Georgia 

Mississippi2 

Pennsylvania4 

Alaska1 

Guam 

Missouri2 

Rhode  Island 

Arizona 

Hawaii 

Montana3 

South  Carolina2 

Arkansas 

Indiana 

Nebraska 

Tennessee 

California 

Iowa 

Nevada1 

Texas 

Colorado 

Kentucky 

New  Hampshire5 

Utah2 

Connecticut 

Louisiana 

New  Jersey 

Vermont 

Delaware 

Maine2 

New  Mexico 

Virginia 

DoDEA/DDESS176 

Maryland 

New  York 

Washington1 

DoDEA/DoDDS1'6 

Massachusetts2 

North  Carolina 

West  Virginia 

District  of  Columbia 

Michigan 

North  Dakota 

Wisconsin 

Florida 

Minnesota 

Oregon3 

Wyoming 

‘Participated  in  the  1996  mathematics  assessment  program  only. 
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Dependent  Elementary  and  Secondary  Schools,  and  DoDDS  is  the  Department  of  Defense  Dependents  Schools. 


The  1996  NAEP  mathematics  assessments  were  based  on  the  same  framework  that  was 
used  to  construct  the  1990  and  1992  assessments.  The  mathematics  framework  and  assessment 
specifications  were  developed  for  NAEP  through  a consensus  project  conducted  by  the  Council 
of  Chief  State  School  Officers  (CCSSO)  under  funding  from  the  National  Assessment  Governing 
Board  (NAGB).  Subsequent  to  the  1992  assessment,  assessment  specifications  were  refined  to 
bring  the  assessment  more  in  line  with  the  Curriculum  and  Evaluation  Standards  for  School 
Mathematics,  published  by  the  National  Council  of  Teachers  of  Mathematics  (NCTM).  Research 
conducted  as  part  of  the  1995  NAEP  field  test  indicated  that  despite  these  specifications,  the 
measurement  constructs  associated  with  the  1992  and  1996  instruments  were  sufficiently  similar 
to  justify  the  continuation  of  the  current  NAEP  scale.  Hence,  for  grade  8,  1996  provides  an 
opportunity  to  report  jurisdiction-level  trend  data  for  a NAEP  mathematics  instrument  for  those 
states  and  territories  that  participated  in  the  1990,  1992,  and  1996  State  Assessment  programs.  In 
addition,  questionnaires  completed  by  the  students,  their  mathematics  teachers,  and  principals  or 
other  school  administrators  provided  an  abundance  of  contextual  data  within  which  to  interpret 
the  mathematics  results. 

The  purpose  of  this  report  is  to  provide  technical  information  about  the  1996  State 
Assessment  in  mathematics.  It  provides  a description  of  the  design  for  the  State  Assessment  and 
gives  an  overview  of  the  steps  involved  in  the  implementation  of  the  program  from  the  planning 
stages  through  to  the  analysis  and  reporting  of  the  data.  The  report  describes  in  detail  the 
development  of  the  cognitive  and  background  questions,  the  field  procedures,  the  creation  of  the 
database  and  data  products  for  analysis,  and  the  methods  and  procedures  used  for  sampling, 
analysis,  and  reporting.  It  does  not  provide  the  results  of  the  assessment— rather,  it  provides 
information  on  how  those  results  were  derived. 
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This  report  is  one  of  several  documents  that  provide  technical  information  about  the 
1996  State  Assessment.  For  those  interested  in  performing  their  own  analyses  of  the  data,  this 
report  and  the  user  guide  for  the  secondary-use  data  should  be  used  as  primary  sources  of 
information  about  NAEP.  Information  for  lay  audiences  is  provided  in  the  procedural  appendices 
to  the  mathematics  subject-area  reports;  theoretical  information  about  the  models  and  procedures 
used  in  NAEP  can  be  found  in  the  special  NAEP-related  issue  of  the  Journal  of  Educational 
Statistics  (Summer  1992/Volume  17,  Number  2). 

Under  a cooperative  agreement  with  the  National  Center  for  Education  Statistics 
(NCES),  Educational  Testing  Service  (ETS)  was  responsible  for  the  development,  analysis,  and 
reporting  of  the  1996  NAEP  programs,  including  the  State  Assessment.  ETS  was  responsible  for 
overall  management  of  aspects  of  the  programs  as  well  as  for  development  of  the  overall  design, 
the  items  and  questionnaires,  data  analysis,  and  reporting.  National  Computer  Systems  (NCS) 
was  a subcontractor  to  ETS  on  both  the  national  and  State  NAEP  programs.  NCS  was 
responsible  for  printing,  distribution,  and  receipt  of  all  assessment  materials,  and  for  data 
processing,  scanning,  and  professional  scoring.  All  aspects  of  sampling  and  field  operations  for 
both  the  national  and  State  Assessments  were  the  responsibility  of  Westat,  Inc.  NCES  contracted 
directly  with  Westat  for  these  services  for  the  national  and  state  assessments. 

This  technical  report  provides  information  about  the  technical  bases  for  a series  of 
reports  that  have  been  prepared  for  the  1996  State  Assessment  program  in  mathematics.  They 
include: 

• A State  Report  for  each  participating  jurisdiction  that  describes  the 
mathematics  scale  scores  of  the  fourth-and  eighth-grade  public-  and 
nonpublic-school  students  in  that  jurisdiction  and  relates  their  scale  scores  to 
contextual  information  about  mathematics  policies  and  instruction. 

• The  NAEP  1996  Mathematics  Report  Card  for  the  Nation  and  the  States, 
which  provides  both  public-  and  nonpublic-school  data  for  major  NAEP 
reporting  subgroups  for  all  of  the  jurisdictions  that  participated  in  the  State 
Assessment  program,  as  well  as  selected  results  from  the  1996  national 
mathematics  assessment. 

• The  Cross-State  Data  Compendium  from  the  NAEP  1996  Mathematics 
Assessment,  which  includes  jurisdiction-level  results  for  all  the  demographic, 
instructional,  and  experiential  background  variables  included  in  the 
Mathematics  Report  Card  and  State  Report. 

• Two  Data  Almanacs  for  each  jurisdiction,  one  for  grade  4 and  one  for  grade 
8,  distributed  only  in  electronic  form,  that  contain  a detailed  breakdown  of 
the  mathematics  scale-score  data  according  to  the  responses  to  the  student, 
teacher,  and  school  questionnaires  for  the  public  school,  nonpublic  school, 
and  combined  populations  as  a whole  and  for  important  subgroups  of  the 
public-school  population.  There  are  six  sections  to  each  almanac: 


=>  7 he  Distribution  Data  Section  provides  information  about  the 
percentages  ot  students  at  or  above  the  three  composite  scale 
achievement  levels  (and  below  basic).  For  the  composite  scale  and 
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each  mathematics  content  strand  scale,2  this  almanac  also  provides 
selected  percentiles  for  the  public  school,  nonpublic  school,  and 
combined  populations  and  for  the  standard  demographic  subgroups 
of  the  public-school  population.  Mathematics  was  previously 
assessed  in  1990  and  1992  for  grade  8 and  in  1992  for  grade  4 in  the 
State  Assessment  program.  For  items  that  are  common  to  1990 
and/or  1992,  trend  results  are  presented,  as  applicable. 

=>  The  Student  Questionnaire  Section  provides  a breakdown  of  the 
composite  scale  score  data  according  to  the  students’  responses  to 
questions  in  the  three  student  questionnaires  included  in  the 
assessment  booklets. 

=>  The  Teacher  Questionnaire  Section  provides  a breakdown  of  the 
composite  scale  score  data  according  to  the  teachers’  responses  to 
questions  in  the  mathematics  teacher  questionnaire. 

=>  The  School  Questionnaire  Section  provides  a breakdown  of  the 
composite  scale  score  data  according  to  the  principals  (or  other 
administrators’)  responses  to  questions  in  the  school  characteristics 
and  policies  questionnaire. 

==>  The  Scale  Section  provides  a breakdown  of  selected  items  from  the 
questionnaires  according  to  each  of  the  scales  measuring 
mathematics  content  strands  in  the  assessment. 


==i>  The  Mathematics  Item  Section  provides  the  response  data  for  each 
mathematics  item  in  the  assessment. 

The  state  reports  and  the  Mathematics  Report  Card  will  be  available  on  the  World  Wide 
Web  as  they  are  publicly  released;  the  almanacs  will  be  placed  on  the  web  about  a month  after 
they  are  released  on  CD-ROM. 


Organization  of  the  Technical  Report 


This  chapter  provides  a description  of  the  design  for  the  State  Assessment  in 
mathematics  and  gives  an  overview  of  the  steps  involved  in  implementing  the  program  from  the 
planning  stages  to  the  analysis  and  reporting  of  the  data.  The  chapter  summarizes  the  major 
components  of  the  program,  with  references  to  later  chapters  for  more  details.  The  organization 
of  this  chapter,  and  of  the  report,  is  as  follows: 

• Section  1.2  provides  an  overview  of  the  design  of  the  1996  State  Assessment 
program  in  mathematics. 


Scales  were  created  for  five  content  strands:  numbei  sense,  properties,  and  operations,  measurement:  geometry  and 
\pati<d  sense;  data  analysts,  statistics,  and  pmbahihty;  and  algebra  and  functions. 
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• Section  1.3  summarizes  the  development  of  the  mathematics  objectives  and 
the  development  and  review  of  the  items  written  to  measure  those  objectives. 
Details  are  provided  in  Chapter  2. 

• Section  1 .4  discusses  the  assignment  of  the  cognitive  items  to  assessment 
booklets.  An  initial  discussion  is  provided  of  the  balanced  incomplete  block 
(BIB)  spiral  design  that  was  used  to  assign  cognitive  items  to  assessment 
booklets  and  assessment  booklets  to  individuals.  A more  complete 
description  is  provided  in  Chapter  2. 

• Section  1.5  outlines  the  sampling  design  used  for  the  1996  State  Assessment 
program  in  mathematics.  A fuller  description  is  provided  in  Chapter  3. 

• Section  1 .6  summarizes  the  field  administration  procedures,  including 
securing  school  cooperation,  training  administrators,  administering  the 
assessment,  and  conducting  quality  control.  Further  details  appear  in 
Chapter  4. 

• Section  1 .7  describes  the  flow  of  the  data  from  their  receipt  at  NCS  through 
data  entry,  professional  scoring,  and  entry  into  the  ETS/NAEP  database  for 
analysis,  and  the  creation  of  data  products  for  secondary  users.  Chapters  5 
and  6 provide  a detailed  description  of  the  process. 

• Section  1.8  provides  an  overview  of  the  data  obtained  from  the  1996  State 
Assessment  program  in  mathematics. 

• Section  1.9  summarizes  the  procedures  used  to  weight  the  assessment  data 
and  to  obtain  estimates  of  the  sampling  variability  of  subpopulation 
estimates.  Chapter  7 provides  a full  description  of  the  weighting  and 
variance  estimation  procedures. 

• Section  l.lw  describes  the  initial  analyses  performed  to  verify  the  quality  of 
the  data  in  preparation  for  more  refined  analyses,  with  details  given  in 
Chapter  9. 

• Section  1.11  describes  the  item  response  theory  (IRT)  scales  and  the  overall 
mathematics  composite  that  were  created  for  the  primary  analysis  of  the 
State  Assessment  data.  Further  discussion  of  the  theory  and  philosophy  of 
the  scaling  technology  appears  in  Chapter  8.  with  details  of  the  scaling 
process  in  Chapter  9. 

• Section  1.12  provides  an  overview  of  the  linking  of  the  scaled  results  from 
the  State  Assessment  to  those  from  the  national  mathematics  assessment. 
Details  of  the  linking  process  appear  in  Chapter  9. 

• Section  1.13  describes  the  reporting  of  the  assessment  results,  with  further 
details  supplied  in  Chapter  10. 
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• Appendices  A through  H include  a list  of  the  participants  in  the  objectives 
and  item  development  process,  a summary  of  the  participation  rates,  a list  ot 
the  conditioning  variables,  the  IRT  parameters  for  the  mathematics  items,  the 
reporting  subgroups,  composite  and  derived  common  background  and 
reporting  variables,  a description  of  the  process  used  to  define  achievement 
levels,  an  explanation  and  correction  of  the  NAHP  program  documentation 
error  in  the  NAHP  1992  results,  and  a summary  explaining  the  information 
weighting  error  in  setting  the  mathematics  achievement  levels. 


1.2  DESIGN  OF  THE  STATE  ASSESSMENT  IN  MATHEMATICS 

The  major  aspects  of  the  design  for  the  State  Assessment  in  mathematics  included  the 
following: 

• Participation  at  the  jurisdiction  ie\cl  was  \oluntary. 

• Fourth-  and  eighth-grade  students  from  public  and  nonpublic  schools  were 
assessed.  Nonpublic  schools  included  Catholic  schools,  other  religious 
schools,  private  schools.  Department  ot  Defense  Education  Activity 
(DoDHA)  schools/  and  BIA  schools.  Separate  representative  samples  of 
public  and  nonpublic  schools  were  selected  in  each  participating  jurisdiction 
and  students  were  randomly  sampled  within  schools.  The  sizes  of  a 
jurisdiction's  nonpublic-school  samples  were  proportional  to  the  percentage 
of  grade-le\ el  students  in  that  jurisdiction  attending  such  schools. 

• The  fourth-  and  eighth-grade  mathematics  student  booklets  used  for  the  1996 
NAHP  State  Assessment,  and  included  as  part  of  the  1996  national  NAHP 
instrument  for  grades  4 and  S,  contained  multiple-choice,  short-constructed 
response,  and  extcndcd-construeted  response  cognitive  items.  Some  items 
required  the  use  of  calculators  (four-function  calculators  at  r ratio  4 and 
scientific  calculators  at  guide  S).  geometric  shapes,  and  proti actors/rulers. 
The  total  pool  of  mathematics  items  was  divided  into  13  blocks  of  items, 
each  15  minutes  long,  at  each  guide  level. 

• A complex  form  of  matrix  sampling  called  a balanced  incomplete  block 
(BIB)  spiraling  design  was  used  With  BIB  spiraling,  students  in  an 
assessment  session  tcccned  different  booklets,  which  provides  lor  greater 
mathematics  content  coverage  than  would  ha\e  been  possible  had  even 
student  been  administered  the  identical  set  of  items,  without  imposing  an 
undue  testing  burden  on  the  student. 

• Background  questionnaires  given  to  the  students,  the  students  mathematics 
teachers,  and  the  principals  or  othei  administrators  provided  a variety  of 
contextual  mimmation  Die  lucKgiomuI  questionnaires  for  the  State 
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Assessment  program  were  identical  to  those  used  in  the  fourth-  and  eighth- 
grade  national  assessments. 

• The  assessment  time  for  each  student  was  approximately  one  hour.  Each 
assessed  student  was  assigned  a mathematics  booklet  that  contained  two 
5-minute  background  questionnaires,  one  3-minute  motivation  questionnaire, 
and  three  of  the  13  blocks  containing  mathematics  items  requiring  15 
minutes  each.  Twenty-six  different  booklets  were  assembled. 

• The  assessments  took  place  in  the  five-week  period  between  January  29  and 
March  4,  1996.  One-fourth  of  the  schools  in  each  jurisdiction  were  to  be 
assessed  each  week  throughout  the  first  four  weeks;  however,  due  to  severe 
weather  throughout  much  of  the  country,  the  fifth  week  was  used  for  regular 
testing  as  well  as  for  makeup  sessions. 

• Dara  collection  was,  by  law,  the  responsibility  of  each  participating 
jurisdiction.  Security  and  uniform  assessment  administration  were  high 
priorities.  Extensive  training  of  State  Assessment  personnel  was  conducted 
to  assure  that  the  assessment  would  be  administered  under  standard,  uniform 
procedures.  For  jurisdictions  that  had  participated  in  previous  NAEP  state 
assessments,  25  percent  of  both  public-  and  nonpublic-school  assessment 
sessions  were  monitored  by  Westat  staff.  For  the  jurisdictions  new  to  NAEP, 
50  percent  of  both  public-  and  nonpublic-school  sessions  were  monitored. 


1 .3  DEVELOPMENT  OF  MATHEMATICS  OBJECTIVES,  ITEMS,  AND 
BACKGROUND  QUESTIONS 

The  same  framework  was  used  for  the  1990,  1992,  and  1996  NAEP  State  Assessments  in 
mathematics.  The  1996  assessment  specifications  represented  an  enhancement  of  the 
specifications  used  for  the  1992  and  1990  that  was  influenced  by  the  National  Council  of 
Teachers  of  Mathematics  (NCTM)  Curriculum  and  Evaluation  Standards  for  School 
Mathematics.  Similar  to  all  previous  NAEP  assessments,  the  mathematics  framework  and 
specifications  were  developed  through  a broad-based  consensus  process.  In  developing  the  new 
portion  of  the  1996  NAEP  mathematics  assessment,  the  same  procedures  used  in  1992  were 
followed,  but  with  the  newly-enhanced  specifications.  Aftercareful  reviews  of  the  framework 
and  specifications,  questions  were  developed  that  were  aligned  with  the  refined  specifications. 

The  framework  for  the  1990,  1992,  and  1996  assessments  was  organized  according  to 
three  mathematical  abilities  and  five  content  strands.  The  mathematical  abilities  assessed  were 
conceptual  understanding , procedural  knowledge , and  problem  solving.  Content  was  drawn 
primarily  from  elementary  and  secondary  school  mathematics  up  to,  but  not  including,  calculus. 
The  content  strands  assessed  were  number  sense , properties , and  operations ; measurement ; 
geometry  and  spatial  sense ; data  analysis , statistics , and  probability ; and  algelra  and  functions* 


4 The  content  strand  number  sense,  properties,  ami  <>/><  ,ation\  was  calU  d numbers  and  operations  in  the  1 900  and 
1992  assessments.  The  content  strand  geometry  and  spatial  sense  was  called  geometn'  in  the  1990  and  1992 
assessments. 
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The  instrument  used  in  the  1996  mathematics  assessment  was  composed  of  a 
combination  of  new  items  developed  for  administration  in  1996  and  items  from  the  1992  and 
1990  assessments.  Those  items  that  were  carried  over  from  the  1992  and  1990  instruments 
comprised  approximately  60  percent  of  the  1996  instrument.  The  remainder  was  made  up  of  new 
items  developed  according  to  the  recommendations  included  in  the  enhanced  framework  for 
1996.  Based  on  field  test  results,  items  that  had  not  been  used  previously  in  a NAEP  assessment 
were  revised  or  modified  as  necessary  and  then  again  reviewed  for  sensitivity,  content,  and 
editorial  concerns.  With  the  assistance  of  ETS/NAEP  staff  and  outside  reviewers,  the 
Mathematics  Instrument  Development  Committee  selected  the  items  to  include  in  the  1996 
assessment.  All  questions  underwent  extensive  reviews  by  specialists  in  mathematics, 
measurement,  and  bias/sensitivity,  as  well  as  reviews  by  state  representatives. 

Maintaining  approximately  70  percent  of  the  instrument  across  the  two  assessment  years 
allowed  for  the  reporting  of  trends  in  mathematics  performance.  At  the  same  time,  developing  a 
new  set  of  items  made  it  possible  to  release  approximately  30  percent  of  the  1992  assessment  for 
public  use.  Copies  of  the  released  items  are  available  from  ETS  after  permission  for  access  has 
been  received  from  NCES. 

Chapter  2 includes  specific  details  about  developing  the  objectives  and  items  tor  the 
State  Assessment. 


1.4  ASSESSMENT  INSTRUMENTS 

The  assembly  of  cognitive  items  into  booklets  and  their  subsequent  assignment  to 
assessed  students  was  determined  by  a BIB  design  with  spiraled  administration.  Details  of  this 
design,  almost  identical  to  the  design  used  in  1990  and  1992,  are  provided  in  Chapter  2.  The 
single  difference  in  these  designs  is  that  the  1996  State  Assessment  does  not  include  blocks  of 
estimation  items.  In  earlier  assessments,  the  results  of  these  blocks  ot  items  were  summarized 
separately  from  the  mathematics  composite  scale.  In  addition  to  the  student  assessment  booklets, 
three  other  instruments  provided  data  relating  to  the  assessment — a mathematics  teacher 
questionnaire,  a school  characteristics  and  policies  questionnaire,  and  an  SD/LEP  student 
questionnaire. 

The  student  assessment  booklets  contained  five  sections  and  included  both  cognitive  and 
noncognitive  questions.  In  addition  to  three  15-minute  sections  of  cognitive  questions,  each 
booklet  included  two  5-minute  sets  of  general  and  mathematics  background  items  designed  to 
gather  contextual  information  about  students,  their  experiences  in  mathematics,  and  their 
attitudes  toward  the  subject,  and  one  3-minute  section  ot  motivation  questions  designed  to  gather 
information  about  the  student's  level  of  motivation  while  taking  the  assessment. 

The  teacher  questionnaire  was  administered  to  the  fourth-  and  eighth-grade  mathematics 
teachers  of  the  students  participating  in  the  assessment.  The  questionnaire  consisted  of  three 
sections  and  took  approximately  20  minutes  to  complete.  The  first  section  focused  on  the 
teacher’s  general  background  and  experience;  the  second,  on  the  teacher's  backgro  md  related  to 
mathematics  and  the  third,  on  classroom  information  about  mathematics  instruction. 

The  school  characteristic s and  policies  questionnaire  was  given  to  the  principal  or  other 
administrator  in  each  participating  school  and  took  about  20  minutes  to  complete.  The  questions 
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asked  about  the  principal’s  background  and  experience;  school  policies,  programs,  and  facilities; 
and  the  demographic  composition  and  background  of  the  students  and  teachers. 

The  SD/LEP  student  questionnaire  was  completed  by  the  teachers  of  those  students  who 
were  selected  to  participate  in  the  State  Assessment  sample  but  who  were  classified  as  students 
with  disabilities  (SD)  or  were  categorized  as  having  limited  English  proficiency  (LEP).  Some  of 
these  students  did  not  participate  in  the  assessment  because  they  were  determined  by  the  school 
personnel  to  be  unable  to  participate,  using  inclusion  rules  provided  by  NAEP;  others  did 
participate  in  the  assessment  because  they  were  determined  to  be  able  to  participate  by  not 
meeting  the  specifications  in  the  inclusion  rules.  Each  questionnaire  took  approximately  three 
minutes  to  complete  and  asked  about  the  student  and  the  special  programs  in  which  the  student 
participated. 

Further  information  on  the  assessment  instruments  can  be  found  in  Chapter  2. 


1 .5  THE  SAMPLING  DESIGN 

The  target  populations  for  the  State  Assessment  program  in  mathematics  consisted  of 
fourth-  and  eighth-grade  students  enrolled  in  public  and  nonpublic  schools.  The  public-  and 
nonpublic-school  samples  in  each  jurisdiction  were  designed  to  produce  aggregate  estimates  for 
the  jurisdiction  and  for  selected  subpopulations  (depending  upon  the  size  and  distribution  of  the 
various  subpopulations  within  the  jurisdiction),  and  also  to  enable  comparisons  to  be  made,  at  the 
jurisdiction  level,  between  administration  of  assessment  tasks  with  monitoring  and  without 
monitoring. 

The  representative  sample  of  public-school  fourth-  and  eighth-grade  students  assessed  in 
the  State  Assessment  came  from  about  100  schools  (per  grade)  in  most  jurisdictions.  However,  if 
a jurisdiction  had  fewer  than  100  schools  with  a fourth  or  eighth  grade,  all  or  almost  all  schools 
were  asked  to  participate.  If  a jurisdiction  had  smaller  numbers  of  students  in  each  school  than 
expected,  more  than  100  schools  were  selected  for  participation.  The  public  schools  were 
stratified  by  urbanization,  percentage  of  Black  and  Hispanic  students  enrolled,  and  median 
household  income  within  the  ZIP  code  area  of  the  school. 

The  nonpublic-school  samples  differed  in  size  across  the  jurisdictions,  with  the  number 
of  schools  selected  proportional  to  the  nonpublic-school  enrollment  within  each  jurisdiction. 
Typically,  about  20  to  25  nonpublic  schools  (per  grade)  were  included  for  each  jurisdiction.  The 
nonpublic  schools  were  stratified  by  type  of  control  (Catholic,  private/other  religious,  other 
nonpublic),  metro  status,  and  enrollment  size  per  grade. 

In  most  jurisdictions,  up  to  30  students  were  selected  from  each  school,  with  the  aim  of 
providing  an  initial  target  sample  size  of  approximately  3,000  public-school  students  per 
jurisdiction.  The  student  sample  size  of  30  for  each  school  was  chosen  to  ensure  that  at  least 
2.000  public-school  students  participated  from  each  jurisdiction  allowing  for  school 
nonresponse,  exclusion  of  students,  inaccuracies  in  the  measures  of  enrollment,  and  student 
absenteeism  from  the  assessment.  In  jurisdictions  with  fewer  schools,  larger  numbers  of  students 
per  school  were  often  required  to  ensure  target  samples  of  roughly  3,000  students.  In  certain 
jurisdictions,  all  eligible  fourth-  and  eighth-grade  students  were  targeted  for  assessment.  The 
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overall  student  sample  size  for  nonpublic  schools  was  much  smaller  than  the  approximate  2,000 
students  from  public  schools  that  were  assessed. 

Students  within  a school  were  sampled  from  lists  of  fourth-  or  eighth-grade  students.  The 
decisions  to  exclude  students  from  the  assessment  were  made  by  school  personnel,  in  one  of  two 
ways,  also  used  in  differing  groups  of  schools  in  the  national  assessment-  The  students  in  one 
group  of  schools  were  excluded  using  the  inclusion  rules  used  in  previous  assessments,  in 
particular,  the  1990  and  1992  Trial  State  Assessments  in  mathematics;  students  in  a second  group 
of  schools  were  excluded  on  the  basis  of  inclusion  rules  that  were  new  for  the  1996  assessment. 
The  new  inclusion  rules  are  meant  to  be  clearer,  more  easily  followed,  and  closer  to  inclusion 
rules  used  in  testing  programs  administered  by  school  districts  or  state  departments  of  education. 
In  the  1996  national  assessments,  students  in  a third  group  of  schools  were  excluded  using  the 
new  inclusion  rules,  but  SD  and  LEP  students  in  these  schools  were  offered  special 
accommodations  to  the  standard  NAEP  administration  procedures.  In  the  State  Assessment,  no 
special  accommodations  were  offered.  Each  excluded  student  in  the  State  Assessment  was 
carefully  accounted  for  to  estimate  the  percentage  of  the  state  population  deemed  unassessable 
and  the  reasons  for  exclusion,  no  matter  which  school  the  student  attended. 

Chapter  3 describes  the  various  aspects  of  selecting  the  sample  for  the  1996  State 
Assessment — selection  of  schools  for  use  of  the  differing  inclusion  criteria,  the  construction  of 
the  public-  and  nonpublic-school  frames,  the  stratification  processes,  the  updating  of  the  school 
frames  with  new  schools,  the  actual  sample  selection,  and  the  sample  selection  for  the  field  test. 


1.6  FIELD  ADMINISTRATION 

The  administration  of  the  1996  program  and  the  1995  field  test  required  collaboration 
between  staff  in  the  participating  jurisdictions  and  schools  and  the  NAEP  contractors,  especially 
Westat,  the  field  administration  contractor.  The  purpose  of  the  field  test  conducted  in  1995  was 
to  try  out  new  blocks  of  items  designed  as  replacements  for  the  1992  assessment  blocks  released 
to  the  public. 

Each  jurisdiction  volunteering  to  participate  in  the  1995  field  test  or  in  the  1996  State 
Assessment  program  was  asked  to  appoint  a state  coordinator  as  liaison  between  NAEP  staff  and 
the  participating  schools.  In  addition,  Westat  hired  and  trained  a supervisor  for  each  jurisdiction 
and  six  field  managers,  each  of  whom  was  assigned  to  work  with  groups  of  jurisdictions.  The 
state  supervisors  were  responsible  for  working  with  the  state  coordinators,  overseeing  assessment 
activities,  training  school  district  personnel  to  administer  the  assessment,  and  coordinating  the 
quality-control  monitoring  efforts.  Each  field  manager  was  responsible  for  working  with  the  state 
coordinators  of  seven  to  eight  jurisdictions  and  supervising  the  state  supervisors  assigned  to 
those  jurisdictions.  An  assessment  administrator  was  responsible  for  preparing  for  and 
conducting  the  assessment  session  in  one  or  more  schools.  These  individuals  were  usually  school 
or  district  staff  and  were  trained  by  Westat.  Westat  also  hired  and  trained  three  to  five  quality 
control  monitors  in  each  jurisdiction.  For  jurisdictions  that  had  previously  participated  in  the 
State  Assessment  program,  25  percent  of  the  public-  and  nonpublic-school  sessions  were 
monitored.  For  jurisdictions  new  to  the  program,  50  percent  of  all  sessions  were  monitored. 
During  the  field  test,  the  state  supervisors  monitored  all  sessions. 
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Chapter  4 describes  the  procedures  for  obtaining  jurisdiction  cooperation  and  provides 
details  about  the  field  activities  for  both  the  field  test  and  1996  State  Assessment  program. 
Chapter  4 also  describes  the  planning  and  preparations  for  the  actual  administration  of  the 
assessment,  the  training  and  monitoring  of  the  assessment  sessions,  and  the  responsibilities  of  the 
state  coordinators,  state  supervisors,  assessment  administrators,  and  quality  control  monitors. 


1 .7  MATERIALS  PROCESSING,  PROFESSIONAL  SCORING,  AND 
DATABASE  CREATION 

Upon  completion  of  each  assessment  session,  school  personnel  shipped  the  assessment 
booklets  and  forms  to  NAEP  contractor  NCS  for  professional  scoring,  entry  into  computer  files, 
and  checking.  The  files  were  then  sent  to  ETS  for  creation  of  the  database.  Chapter  5 describes 
the  printing,  distribution,  receipt,  processing,  and  final  disposition  of  the  1996  State  Assessment 
materials. 

The  volume  of  collected  data  and  the  complexity  of  the  State  Assessment  processing 
design,  with  its  spiraled  distribution  of  booklets,  as  well  as  the  concurrent  administration  of  this 
assessment  and  the  national  assessments,  required  the  development  and  implementation  of 
flexible,  innovative  processing  programs,  and  a sophisticated  Process  Control  System.  This 
system,  described  in  Chapter  5,  allowed  an  integration  of  data  entry  and  workflow  management 
systems  that  included  carefully  planned  and  delineated  editing,  quality  control,  and  auditing 
procedures. 

Chapter  5 also  describes  the  data  transcription  and  editing  procedures  used  to  generate 
the  electronic  files  containing  various  assessment  information,  including  the  sampling  weights 
required  to  make  valid  statistical  inferences  about  the  population  from  which  the  State 
Assessment  sample  was  drawn.  Before  any  analysis  could  begin,  the  data  from  these  files 
underwent  a quality  control  check  at  ETS.  The  files  were  then  merged  into  a comprehensive, 
integrated  database.  Chapter  6 describes  the  transcribed  data  files,  the  procedure  of  merging  them 
to  create  the  State  Assessment  database,  the  results  of  the  quality  control  process,  and  the 
procedures  used  to  create  data  products  for  use  in  secondary  research. 


1.8  THE  1996  STATE  ASSESSMENT  DATA 

The  basic  information  collected  from  the  State  Assessment  in  mathematics  consisted  of 
the  responses  of  the  assessed  students  to  the  144  mathematics  exercises  at  grade  4 and  164 
exercises  at  grade  8.  To  limit  the  assessment  time  for  each  student  to  about  one  hour,  a variant  of 
matrix  sampling  called  BIB  spiraling  was  used  to  assign  a subset  of  the  full  exercise  pool  to  each 
student.  At  each  grade  level,  the  set  of  items  was  divided  into  13  unique  blocks,  each  requiring 
15  minutes  for  completion.  Each  assessed  student  received  a booklet  containing  three  of  the  13 
blocks  according  to  a design  that  ensured  that  each  block  was  administered  to  a representative 
sample  of  students  within  each  jurisdiction.  The  data  also  included  responses  to  the  background 
questionnaires  (described  in  wSection  1.4  of  this  chapter  and  in  Chapter  2). 

The  national  data  to  which  the  State  Assessment  results  were  compared  came  from 
nationally  representative  samples  of  public-  and  nonpublic-school  students  in  the  fourth  and 
eighth  grade.  These  samples  were  part  of  the  full  1996  national  mathematics  assessment  in  which 


nationally  representative  samples  of  students  in  public  and  nonpublic  schools  were  assessed  from 
three  age  cohorts:  fourth-,  eighth-,  and  twelfth-grade  students. 

The  assessment  instruments  used  in  the  State  Assessment  were  also  used  in  the  fourth- 
and  eighth-grade  national  assessments  and  were  administered  using  almost  identical  procedures 
in  both  assessments.  The  time  of  testing  for  the  state  assessments  (January  29-March  4,  1996) 
occurred  within  the  time  of  testing  of  the  national  assessment  (January  3-ApriI  5,  1996). 
However,  the  state  assessments  differed  from  the  national  assessment  in  one  important  regard: 
Westat  staff  collected  the  data  for  the  national  assessment  while,  in  accordance  with  the  NAEP 
legislation,  data  collection  activities  for  the  State  Assessment  were  the  responsibility  of  each 
participating  jurisdiction.  These  activities  included  ensuring  the  participation  of  selected  schools 
and  students,  assessing  students  according  to  standardized  procedures,  and  observing  procedures 
for  test  security.  To  provide  quality  control  of  the  State  Assessment,  a random  half  of  the 
administrations  in  jurisdictions  participating  in  a State  Assessment  for  the  first  time  was 
monitored;  25  percent  of  the  administrations  in  other  jurisdictions  was  monitored. 


1.9  WEIGHTING  AND  VARIANCE  ESTIMATION 

A complex  sample  design  was  used  to  select  the  students  to  be  assessed  in  each  of  the 
participating  jurisdictions.  The  properties  of  a sample  from  a complex  design  are  very  different 
from  those  of  a simple  random  sample  in  which  every  student  in  the  target  population  has  an 
equal  chance  of  selection  and  every  combination  of  students  of  the  size  of  the  sample  has  an 
equal  chance  of  selection.  The  properties  of  the  sample  from  the  complex  State  Assessment 
design  were  taken  into  account  in  the  analysis  of  the  assessment  data. 

One  way  that  the  properties  of  the  sample  design  were  addressed  was  by  using  sampling 
weights  to  account  for  the  fact  that  the  probabilities  of  selection  were  not  identical  for  all 
students.  These  weights  also  included  adjustments  for  school  and  student  nonresponse.  All 
population  and  subpopulation  characteristics  based  on  the  State  Assessment  data  used  sampling 
weights  in  their  estimation.  Chapter  7 provides  details  on  the  computation  of  these  weights. 

In  addition  to  deriving  appropriate  estimates  of  population  characteristics,  it  is  essential 
to  obtain  appropriate  measures  of  the  degree  of  uncertainty  of  those  statistics.  One  component  of 
uncertainty  is  a result  of  sampling  variability,  which  measures  the  dependence  of  the  results  on 
the  particular  sample  of  students  actually  assessed.  Because  of  the  effects  of  cluster  selection 
(schools  are  selected  first,  then  students  are  selected  within  those  schools),  observations  made  on 
different  students  cannot  be  assumed  to  be  independent  of  each  other  (and,  in  fact,  are  generally 
positively  correlated).  As  a result,  classical  variance  estimation  formulas  will  produce  incorrect 
results.  Instead,  a variance  estimation  procedure  that  takes  the  characteristics  of  the  sample  into 
account  was  used  for  all  analyses.  This  procedure,  called  jackknife  variance  estimation , is 
discussed  in  Chapter  7 and  described  more  fully  in  The  NAEP  1994  Technical  Report  (Allen, 
Kline,  & Zelenak,  1996). 

Jackknife  variance  estimation  provides  a reasonable  measure  of  uncertainty  for  any 
statistic  based  on  values  observed  without  error.  Statistics  such  as  the  average  proportion  of 
students  correctly  answering  a given  question  meet  this  requirement,  but  other  statistics  based  on 
estimates  of  student  mathematics  performance,  such  as  the  average  mathematics  scale  score  of  a 
subpopulation,  do  not.  Because  each  student  typically  responds  to  relatively  few  items  within  a 
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particular  mathematics  content  strand,  there  exists  a nontrivial  amount  of  imprecision  in  the 
measurement  of  the  proficiency  of  a given  student.  This  imprecision  adds  an  additional 
component  of  variability  to  statistics  based  on  estimates  of  individual  scale  scores.  The 
estimation  of  this  component  of  variabi i it\  is  discussed  in  Chapter  X. 


1.10  PRELIMINARY  DATA  ANALYSIS 

After  the  computer  files  of  student  responses  were  received  from  NCS,  all  cognitive  and 
noncognitive  items  were  subjected  to  an  extensive  item  analysis.  Haeh  block  of  cognitive  items 
was  subjected  to  item  analysis  routines,  which  yielded  for  each  item  the  number  of  respondents, 
the  percentage  of  responses  in  each  response  category  for  an  item,  the  percentage  w ho  omitted 
the  item,  the  percentage  w ho  did  not  reach  the  item,  and  the  correlation  between  the  item  score 
and  the  item  block  score.  In  addition,  the  item  analysis  program  provided  summary  statistics  for 
each  block,  including  a reliability  (internal  consistency  > coefficient.  These  analyses  w'ere  used  to 
check  on  the  scoring  of  the  items,  to  verify  the  appropriateness  of  the  difficulty  level  of  the 
items,  and  to  check  for  speededness.  The  results  also  were  reviewed  by  knowledgeable  project 
staff  in  search  of  aberrations  that  might  signal  unusual  results  or  errors  in  the  database. 

'fables  of  the  weighted  percentages  of  students  with  responses  in  each  category'  of  each 
cognitive  and  background  item  were  created  and  distributed  to  each  jurisdiction.  Additional 
analyses  comparing  the  data  from  the  monitored  sessions  with  those  from  the  unmonitored 
sessions  were  conducted  to  determine  the  comparability  of  the  assessment  data  from  the  two 
types  of  administrations.  Differential  item  functioning  (DIPT  analyses  were  carried  out  to  identify 
items  new  to  the  assessment  that  were  differentially  difficult  for  yarious  subgroups  and  to 
reexamine  such  items  yvith  respect  to  their  fairness  and  their  appropriateness  lor  inclusion  in  the 
scaling  process,  further  details  of  the  preliminary  analyses  conducted  on  the  data  appear  in 
Chapter  9. 


LI  1 SCALING  THE  ASSESSMENT  H EMS 


The  primary  analysis  and  reporting  o!  the  icsults  from  the  State  Assessment  program 
used  item  response  theory  (IRT)  scale-score  models.  Scaling  models  quantify  a respondent’s 
tendency  to  provide  correct  answers  to  the  domain  of  items  contributing  to  a scale  as  a function 
of  a parameter  called  proficiency,  estimated  by  a scale  score.  The  scale  scores  can  be  viewed  as  a 
summary  measure  of  performance  across  the  domain  of  items  that  make  up  the  scale.  Three 
distinct  IRT  models  were  used  for  scaling:  I » three-parameter  logistic  models  for  multiple-choice 
items;  2)  two-parameter  logistic  models  for  short  constructed-response  items  that  were  scored 
correct  or  incorrect,  and  3)  generalized  pamal-ered.it  models  for  short  and  extended  constructed- 
response  items  that  were  scored  on  a multipoint  scale.  Chapter  X provides  an  overview  of  the 
scaling  models  used,  further  details  on  the  application  of  these  models  are  provided  in  Chapter  9. 

A series  of  scales  were  created  lor  the  State  Assessment  to  summarize  students' 
mathematics  performance.  These  scales  were  defined  identically  to  those  used  lor  the  scaling  ot 
the  national  NAHP  fourth-  and  eighth-grade  mathematics  data,  five  content  strand  scales,  based 
on  the  paradigm  described  in  Chapter  2.  were  created  to  eoriespond  to  the  following  areas: 
rumhrr  sense,  properties,  and  operations,  nh'asnri  inent:  aeowett  \ and  spatial  sense's  data 
aiud\  s/s.  stat i \t i(  s,  and  prohahiht  \ , and  alvt  lna  .///./  fmn  fioir  Although  the  items  comprising 


each  scale  were  identical  to  those  used  for  the  national  program,  the  item  parameters  for  the  State 
Assessment  scales  were  estimated  from  the  combined  data  from  all  jurisdictions  participating  in 
the  State  Assessment.  Item  parameter  estimation  was  based  on  an  item  calibration  sample 
consisting  of  an  approximately  25  percent  sample  of  all  the  available  data.  To  ensure  equal 
representation  in  the  scaling  process,  each  jurisdiction  was  equally  represented  in  the  item 
calibration  sample,  as  were  the  monitored  and  unmonitored  administrations  from  each 
jurisdiction.  Chapter  9 provides  further  details  about  the  item  parameter  estimation. 

The  fit  of  the  IRT  model  to  the  observed  data  was  examined  within  each  scale  by 
comparing  the  estimates  of  the  empirical  item  characteristic  functions  with  the  theoretic  curves. 
For  multiple-choice  and  dichotomously-scored  constructed  response  items,  nonmodel-based 
estimates  of  the  expected  proportions  of  correct  responses  to  each  item  for  students  with  various 
levels  of  scale  scores  were  compared  with  the  Fitted  item  response  curve;  for  partial-credit 
polytomously-scored  constructed-response  items,  the  comparisons  were  based  on  the  expected 
proportions  of  students  with  various  levels  of  scale  scores  who  achieved  each  item  score  level.  In 
general,  the  item-level  results  were  well  fit  by  the  scaling  models. 

Using  the  item  parameter  estimates,  estimates  of  various  population  statistics  were 
obtained  for  each  jurisdiction.  The  NAEP  methods  use  random  draws  (“plausible  values”)  from 
scale  score  distributions  for  each  student  to  compute  population  statistics.  Plausible  values  are 
not  optimal  individual  student  scale  scores;  instead,  they  serve  as  intermediate  values  to  be  used 
in  estimating  population  characteristics.  Under  the  assumptions  of  the  scaling  models,  these 
population  estimates  will  be  consistent,  in  the  sense  that  the  estimates  approach  the  model-based 
population  values  as  the  sample  size  increases.  This  would  not  be  the  case  for  population 
estimates  obtained  by  aggregating  optimal  individual  scale  scores.  Chapter  8 provides  further 
details  on  the  computation  and  use  of  plausible  values. 

In  addition  to  the  plausible  values  for  each  scale,  a composite  score  scale  of  the  five 
mathematics  content  strand  scales  was  created  as  a measure  of  overall  mathematics  proficiency. 
This  composite  was  a weighted  average  of  the  five  mathematics  content  strand  scales  in  which 
the  weights  were  proportional  to  the  relative  importance  assigned  to  each  content  strand  as 
specified  in  the  mathematics  objectives.  Consistent  with  the  mathematics  framework,  the  weights 
used  to  define  the  composite  were  somewhat  different  at  grades  4 and  8.  The  definitions  of  the 
composites  for  the  State  Assessment  program  at  grades  4 and  8 were  identical  to  those  used  for 
the  national  fourth-  and  eighth-grade  mathematics  assessments. 


M2  LINKING  THE  STATE  RESULTS  TO  THE  NATIONAL  RESULTS 

A major  purpose  of  the  State  Assessment  program  was  to  allow  each  participating 
jurisdiction  to  compare  its  1996  results  with  the  nation  as  a whole  and  with  the  region  of  the 
country  in  which  that  jurisdiction  is  located.  For  meaningful  comparisons  to  be  made  between 
each  of  the  State  Assessment  jurisdictions  and  the  relevant  national  sample,  results  from  these 
two  assessments  had  to  be  expressed  in  terms  of  a similar  system  of  scale  units. 

The  results  from  the  State  Assessment  program  were  linked  to  those  from  the  national 
assessment  through  linking  functions  determined  by  comparing  the  results  for  the  aggregate  of  all 
fourth-  and  eighth-grade  public-school  students  assessed  in  the  State  Assessment  with  the  results 
for  public-school  students  of  the  matching  grade  within  a subsample  (the  National  Linking 
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sample)  of  the  national  NAEP  sample.  The  National  Linking  sample  for  a given  grade  is  a 
representative  sample  of  the  population  of  all  grade-eligible  public-school  students  within  the 
aggregate  of  the  45  participating  states  and  the  District  of  Columbia  (excluding  Guam  and  the 
two  DoDEA  jurisdictions).  Specifically,  the  grade  4 National  Linking  sample  consists  of  all 
fourth-grade  students  in  public  schools  in  the  states  and  the  District  of  Columbia  who  were 
assessed  in  the  national  mathematics  assessment.  The  grade  8 National  Linking  sample  is 
equivalently  defined  for  eighth-grade  students  who  participated  in  the  national  assessment. 

For  each  grade,  a linear  equating  within  each  scale  was  used  to  link  the  results  of  the 
State  Assessment  to  the  national  assessment.  For  each  scale,  the  adequacy  of  linear  equating  was 
evaluated  by  comparing  the  distribution  of  mathematics  scale  scores  based  on  the  aggregation  of 
all  assessed  students  at  each  grade  from  the  participating  states  and  the  District  of  Columbia  with 
the  equivalent  distribution  based  on  the  students  in  the  National  Linking  sample  for  the  matching 
grade.  In  the  estimation  of  these  distributions,  the  students  were  weighted  to  represent  the  target 
population  of  public-school  students  in  the  specified  grade  in  the  aggregation  of  the  states  and 
the  District  of  Columbia.  If  a linear  equating  was  adequate,  the  distribution  for  the  aggregate  of 
states  and  the  District  of  Columbia  and  that  for  the  National  Linking  sample  would  have,  to  a 
close  approximation,  the  same  shape,  in  terms  of  the  skewness,  kurtosis,  and  higher  moments  of 
the  distributions.  The  only  differences  in  the  distributions  allowed  by  linear  equating  are  in  the 
means  and  variances.  This  has  been  found  to  be  the  case  for  the  1996  State  Assessment  program. 

Each  mathematics  content  strand  scale  was  linked  by  matching  the  mean  and  standard 
deviation  of  the  scale  score  averages  across  all  fourth-  or  eighth-grade  students  in  the  State 
Assessment  to  the  corresponding  scale  mean  and  standard  deviation  across  all  students  in  the 
matching  grade  National  Linking  sample.  The  results  for  nonpublic-school  students  were 
transformed  to  the  national  scale  using  the  same  transformation  as  was  used  for  the  public-school 
student  results.  Further  details  of  the  linking  are  given  in  Chapter  9. 


1.13  REPORTING  THE  STATE  ASSESSMENT  RESULTS 

Each  jurisdiction  in  the  State  Assessment  received  a summary  report  providing  its  results 
with  accompanying  text  and  tables,  national  and  regional  comparisons,  and  (for  jurisdictions  that 
had  participated  in  the  1990  and  1992  state  programs)  trend  comparisons  to  the  previous 
assessments.  These  reports  were  generated  by  a computerized  report-generation  system  for  which 
graphic  designers,  statisticians,  data  analysts,  and  report  writers  collaborated  to  develop  shells  of 
the  reports  in  advance  of  the  analysis.  These  prototype  reports  were  provided  to  State  Education 
Agency  personnel  for  their  reviews  and  comments.  The  results  ot  the  data  analysis  were  then 
automatically  incorporated  into  the  reports,  which  display  tables  and  graphs  of  the  results  and 
interpretations  of  those  results,  including  indications  of  subpopulation  comparisons  of  statistical 
and  substantive  significance. 


Each  report  contains  state-level  estimates  of  average  scale  score,  both  for  the  state  as  a 
whole  and  for  categories  of  the  key  reporting  variables:  gender,  race/ethnicity,  level  of  parental 
education,  and  type  of  location.  Results  are  presented  for  each  mathematics  scale  score,  for  the 
overall  mathematics  composite  scale  score,  and  by  achievement  levels.  Results  are  also  reported 
for  a variety  of  other  subpopulations  based  on  variables  derived  from  the  student,  teacher,  and 
school  questionnaires.  Standard  errors  are  included  for  all  statistics. 


A second  report,  the  NAEP  1996  Mathematics  Report  Card  for  the  Nation  and  the  States , 
highlights  key  assessment  results  for  the  nation  and  summarizes  results  across  the  jurisdictions 
participating  in  the  assessment.  This  report  contains  composite  scale-score  results  (scale-score 
means,  proportions  at  or  above  achievement  levels,  etc.)  for  the  nation,  each  of  the  four  regions 
of  the  country,  and  each  jurisdiction  participating  in  the  State  Assessment,  both  overall  and  by 
the  primary  reporting  variables.  In  addition,  overall  results  are  reported  for  each  of  the  content 
strand  scales. 


The  third  type  of  summary  report  is  entitled  Cross-State  Data  Compendium  from  the 
NAEP  1996  Mathematics  Assessment.  Like  the  Report  Card,  the  Compendium  reports  results  for 
the  nation  and  for  all  of  the  jurisdictions  participating  in  the  State  Assessment.  The  Compendium 
contains  most  of  the  tables  included  in  the  Report  Card  plus  additional  tables  that  provide 
composite  scale-score  results  for  a large  number  of  secondary  reporting  variables. 

The  fourth  type  of  summary  report  is  a six-section  almanac.  One  section  of  the  almanac 
includes  information  about  the  percentages  of  students  at  or  above  the  three  composite  scale 
achievement  levels  (and  below  basic).  Three  of  the  sections  of  the  almanac  present  analyses 
based  on  responses  to  each  of  the  questionnaires  (student,  mathematics  teacher,  and  school) 
administered  as  part  of  the  State  Assessment.  Another  section  of  the  almanac,  the  scale  section, 
reports  scale  score  means  and  associated  standard  errors  for  the  five  mathematics  content  strand 
scales.  Results  in  this  section  are  also  reported  for  the  total  group  in  each  jurisdiction,  as  well  as 
for  select  subgroups  of  interest.  The  final  section  of  the  almanac,  the  “p-value”  section,  provides 
the  total-group  proportion  of  correct  responses  to  each  cognitive  item  included  in  the  assessment. 

The  production  of  the  state  reports.  Mathematics  Report  Card,  Data  Compendium , and 
the  almanacs  required  a large  number  of  decisions  about  a variety  of  data  analysis  and  statistical 
issues.  For  example,  because  the  demographic  characteristics  of  the  fourth-  and  eighth-grade 
public-school  students  vary  widely  by  jurisdiction,  the  proportions  of  students  in  the  various 
categories  of  the  race/ethnicity,  parental  education,  and  type  of  location  variables  also  varied  by 
jurisdiction.  Chapter  10  documents  the  major  conventions  and  statistical  procedures  used  in 
generating  the  state  reports.  Mathematics  Report  Card , Data  Compendium,  and  the  almanacs. 

The  chapter  describes  the  rules,  based  on  effect  size  and  sample  size  considerations,  that  were 
used  to  establish  whether  a particular  category  contained  sufficient  data  for  reliable  reporting  of 
results  for  a particular  jurisdiction.  Chapter  10  also  describes  the  multiple  comparison  and  effect 
size-based  inferential  rules  that  were  used  for  evaluating  the  statistical  and  substantive 
significance  of  subpopulation  comparisons. 

To  provide  information  about  the  general izability  of  the  results,  a variety  of  information 
about  participation  rates  was  reported  for  each  state  and  jurisdiction.  This  included  the  school 
participation  rates,  both  in  terms  of  the  initially  selected  samples  of  schools  and  in  terms  of  the 
finally  achieved  samples,  including  replacement  schools.  The  student  participation  rates,  the 
rates  of  students  excluded  due  to  being  identified  as  SD  or  LEP,  and  the  estimated  proportions  of 
assessed  students  w'ho  are  classified  as  SD  or  LEP  were  also  reported  by  jurisdiction.  These  rates 
are  described  and  reported  in  Appendix  B. 
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Chapter  2 


DEVELOPING  THE  MATHEMATICS  OBJECTIVES,  COGNITIVE 
ITEMS,  BACKGROUND  QUESTIONS,  AND  ASSESSMENT 

INSTRUMENTS1 

Jeff  Haberstroh  and  Chancey  Jones 
Educational  Testing  Service 


2.1  OVERVIEW 

A new  framework  was  developed  for  the  1996  NAEP  State  Assessment  in  mathematics. 
This  framework  represents  an  enhancement  of  the  framework  used  for  the  1992  and  1990 
assessments.  Similar  to  all  previous  NAEP  assessments,  the  mathematics  framework  was 
developed  through  a broad-based  consensus  process.  The  National  Assessment  Governing  Board 
(NAGB)  contracted  with  The  College  Board  to  prepare  the  framework  for  the  1996  mathematics 
assessment.  The  development  process  involved  a committee  of  mathematicians  and  mathematics 
educators.  Educators,  scholars,  and  citizens,  representative  of  many  diverse  constituencies  and 
points  of  view,  participated  in  the  national  consensus  process  to  design  objectives  for  the 
assessment. 

The  instrument  used  in  the  1996  mathematics  assessment  was  composed  of  a 
combination  of  new  items  developed  for  administration  in  1996  and  items  from  the  1992  and 
1990  assessments.  Those  items  that  were  carried  over  from  the  1992  and  1990  instruments 
comprised  approximately  60  percent  of  the  1996  instrument.  The  remaining  portion  was  made  up 
of  new  items  developed  according  to  the  recommendations  included  in  the  enhanced 
specifications  for  the  1996  framework.  Maintaining  approximately  60  percent  of  the  instrument 
across  the  two  assessment  years  (1992  and  1996)  allowed  for  the  reporting  of  trends  in 
mathematics  performance.  At  the  same  time,  developing  a new  set  of  items  made  it  possible  to 
release  approximately  40  percent  of  the  1992  assessment  for  public  use. 

In  developing  the  new  portion  of  the  1996  NAEP  mathematics  assessment,  the  same 
procedures  used  in  1992  were  followed,  but  using  the  newly  enhanced  specifications  from  the 
framework.  After  careful  reviews  of  the  framework,  items  were  developed  that  were  aligned  with 
the  specifications  described  in  the  framework.  All  items  underwent  extensive  reviews  by 
specialists  in  mathematics,  measurement,  and  bias/sensitivity,  as  well  as  reviews  by  state 
representatives. 

The  framework  and  item  development  efforts  were  governed  by  four  major 
considerations: 

• The  framework  for  the  mathematics  assessment  had  to  be  developed  through 
a consensus  process. 


1 Jeff  Haberstroh  and  Chanccy  Jones  coordinated  the  development  of  the  mathematics  assessment  instruments. 
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• As  outlined  in  the  ETS  proposal  for  the  administration  of  the  NAEP 
cooperative  agreement  (ETS,  1992),  the  development  of  the  items  had  to  be 
guided  by  a Mathematics  Instrument  Development  Panel  and  receive  further 
review  by  state  representatives  and  classroom  teachers  from  across  the 
country.  In  addition,  the  items  had  to  be  carefully  reviewed  for  potential 
bias. 

• As  described  in  the  ETS  Standards  of  Quality  and  Fairness  (ETS,  1987),  all 
materials  developed  at  ETS  had  to  be  in  compliance  with  specified 
procedures. 

• As  per  federal  regulations,  all  NAEP  cognitive  and  background  items  had  to 
be  submitted  to  a federal  clearance  process. 

This  chapter  includes  details  about  developing  the  specifications  and  items  for  the  State 
Assessment  in  mathematics.  The  chapter  also  describes  the  instruments,  the  student  assessment 
booklets,  mathematics  teacher  questionnaire,  school  characteristics  and  policies  questionnaire, 
and  SD/LEP  student  questionnaire.  Various  committees  worked  on  the  development  of  the 
framework,  objectives,  and  items  for  the  mathematics  assessment.  The  list  of  committee 
members  and  consultants  who  participated  in  the  1996  development  process  is  provided  in 
Appendix  A. 


2.2  FRAMEWORK  AND  ASSESSMENT  DESIGN  PRINCIPLES 

The  guidelines  for  the  development  of  the  new  mathematics  framework  specified  that  the 
design  of  the  framework  contain  some  performance-oriented  exercises  in  mathematics  with  a 
focus  on  problem-solving  and  providing  students  with  opportunities  to  communicate  their 
understanding  in  mathematics.  The  framework  would  embody  a broad  view  of  mathematics  that 
addressed  the  high  levels  of  mathematical  literacy  needed  for  employability,  personal 
development,  and  citizenship.  Also,  the  framework  would  take  into  account  findings  of 
contemporary  research  on  mathematics  and  mathematics  education,  and  would  expand  the  range 
of  assessment  tools  to  include  formats  that  more  closely  resembled  desired  classroom  activities. 

The  development  was  further  guided  by  the  consideration  that  the  assessment  should 
reflect  many  of  the  states’  curricular  emphases  and  objectives  in  addition  to  what  various 
scholars,  practitioners,  and  interested  citizens  believed  should  be  included  in  the  curriculum. 
Accordingly,  the  committee  focused  on  several  frames  of  reference: 

• The  purpose  of  the  NAEP  mathematics  assessment  is  to  provide  information 
about  the  progress  and  achievement  of  students  in  general  rather  than  to  test 
individual  students’  ability.  NAEP  is  designed  to  inform  policymakers  and 
the  public  about  mathematics  ability  in  the  United  States.  Furthermore, 

NAEP  state  data  can  be  used  to  inform  states  of  their  students’  relative 
strengths  and  weaknesses. 

• The  term  “mathematical  literacy"  encompasses  such  broad  skills  and 
abilities  as  being  able  to  reason  numerically,  algebraically,  geometrically, 
spatially,  and  with  data;  identify  and  apply  problem-solving  strategics 
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appropriately  in  situations;  and  use  the  language  of  mathematics  to  construct 
clear  and  coherent  responses  to  problems  or  tasks. 

• The  mathematics  assessment  should  use  authentic  problems  and  tasks  that 
address  important  mathematics  concepts  and  skills  so  that  the  assessment 
tool  will  demonstrate  a close  link  to  desired  classroom  instruction  and 
students’  mathematics  experiences. 

• Every  effort  should  be  made  to  make  the  best  use  of  available  methodology 
and  resources  in  driving  assessment  capabilities  forward. 

• Every  effort  must  be  made  in  developing  the  assessment  to  represent  a 
variety  of  opinions,  perspectives,  and  emphases  among  professionals  in 
universities,  as  well  as  in  state  and  local  school  districts. 


23  FRAMEWORK  DEVELOPMENT  PROCESS 

NAGB  is  responsible  for  guiding  NAEP,  including  the  development  of  the  mathematics 
assessment  objectives  and  test  specifications.  Appointed  by  the  Secretary  of  Education  from  lists 
of  nominees  proposed  by  the  board  itself  in  various  statutory  categories,  the  24-member  board  is 
composed  of  state,  local,  and  federal  officials,  as  well  as  educators  and  members  of  the  public. 

Under  contract  with  NAGB,  The  College  Board  convened  a committee  during  1991  and 
1 992  to  develop  an  enhanced  version  of  the  framework  that  had  been  used  for  the  1990  and  1992 
assessments.  (Note  that  the  1992  mathematics  assessment  had  already  been  developed  at  the  time 
the  development  of  this  enhanced  framework  was  just  beginning.)  The  enhanced  version  was 
needed  to  better  reflect  the  rapid  evolution  of  mathematics  instruction  that  was  underway  in  the 
early  1 990s  as  a result  of  the  emergence  of  the  National  Council  of  Teachers  of  Mathematics 
Curriculum  and  Evaluation  Standards  for  School  Mathematics  (NCTM  Standards ).  The 
Standards  were  rapidly  becoming  one  of  the  acknowledged  barometers  for  measuring 
achievement  in  mathematics  education.  The  development  process  for  the  enhanced  framework 
was  based  on  consensus  building,  and  included  the  committee  listed  in  Appendix  A. 

During  this  development  process,  input  and  reactions  were  continually  sought  from  a 
wide  range  of  members  of  the  mathematics  field,  experts  in  assessment,  school  administrators, 
and  state  staff  in  mathematics  assessment.  In  particular,  the  process  was  informed  by 
recommendations  of  leading  professional  organizations  in  mathematics. 


2.4  FRAMEWORK  FOR  THE  ASSESSMENT 

The  framework  for  the  1996  mathematics  assessment  is  organized  according  to  a five-by- 
three  matrix  of  content  strands  by  mathematical  abilities.  The  content  strands  are: 

• Number  Sense,  Properties,  and  Operations; 

• Measurement; 

• Geometry  and  Spatial  Sense; 
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Data  Analysis,  Statistics,  and  Probability;  and 
Algebra  and  Functions. 


These  content  strands  were  assessed  across  the  three  mathematical  abilities  of: 

• Conceptual  Understanding, 

• Procedural  Knowledge,  and 

• Problem  Solving. 

While  most  questions  in  the  assessment  assessed  understanding  of  a single  subtopic  in  a 
content  strand  and  a single  mathematical  ability,  some  questions  assessed  multiple  content 
subtopics,  strands,  and/or  multiple  mathematical  abilities.  Figures  2-1  and  2-2  describe  the  Five 
content  strands  and  three  mathematical  abilities  that  guided  the  development  of  the  1996  State 
Assessment  program  in  mathematics. 
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Figure  2-1 

\ Descriptions  of  Content  Strands  in  Mathematics 


Number  Sense , Propetyies,  and  Operations 

\ \ 

This  strand  focuses  on  students’  understanding  of  numbers  (whole\ numbers,  fractions, 
decimals,  integers,  real  numbers,  and  complex  numbers),  operations,  and  estimation,  and  their 
application  to  real-world  situations.  Students  will  be  expected  to  demonstrate  an  understanding 
of  numerical  relationships  as  expressed  in  ratios,  proportions,  and  percents.  Students  also  will  be 
expected  to  understand  properties  of  numbers  and  operations,  generalize  from  number  patterns, 
and  verify  results. 


Measurement 

The  measurement  strand  focuses  on  understanding  of  the  process  of  measurement  and  on 
the  use  of  numbers  and  measures  to  describe  and  compare  mathematical  and  real-world  objects. 
Students  will  be  asked  to  identify  attributes,  select  appropriate  units  and  tools,  apply 
measurement  concepts,  and  communicate  measurement-related  ideas. 


Geometry  d^d  Spatial  Sense 

As  described  in  the  NCTM  Standards , spatial  sense  must  be  an  integral  component  of  the 
study  and  assessment  of  geometry.  Understanding  spatial  relationships  allows  students  to  use  the 
dynamic  nature  of  geometry  to  connect  mathematics  to  their  world. 

This  content  strand  is  designed  to  extend  well  beyond  low-level  identification  of 
geometric  shapes  into  transformations  and  combinations  of  those  shapes.  Informal  constructions 
and  demonstrations  (including  drawing  representations),  along  with  their  justifications,  take 
precedence  over  more  traditional  types  of  compass-and-straightedge  constructions  and  proofs. 
While  reasoning  is  addressed  throughout  all  of  the  content  strands,  this  strand  continues  to  lend 
itself  to  the  demonstration  of  reasoning  within  both  formal  and  informal  settings.  The  extension 
of  proportional  thinking  to  similar  figures  and  indirect  measurement  is  an  important  connection 
here. 


Data  Analysis , Statistics , and  Probability 

The  important  skills  of  collecting,  organizing,  reading,  representing,  and  interpreting 
data  will  be  assessed  in  a variety  of  contexts  to  reflect  the  pervasive  use  of  these  skills  in  dealing 
with  information.  Statistics  and  statistical  concepts  extend  these  basic  skills  to  include  analyzing 
and  communicating  increasingly  sophisticated  interpretations  of  data.  Dealing  with  uncertainty 
and  making  predictions  about  outcomes  require  an  understanding  not  only  of  the  meaning  of 
basic  probability  concepts  hut  also  the  application  of  those  concepts  in  problem-solving  and 
decision-making  situations. 


Figure  2-1  (continued) 

Descriptions  of  Content  Strands  in  Mathematics 


Questions  will  emphasize  appropriate  methods  for  gathering  data,  the  visual  exploration 
of  data,  a variety  of  ways  of  representing  data,  and  the  development  and  evaluation  of  arguments 
based  on  data  analysis.  Students  will  be  expected  to  apply  these  ideas  in  increasingly 
sophisticated  situations  that  require  increasingly  comprehensive  analysis  and  decision  making. 

Algebra  and  Functions 

This  strand  extends  from  work  with  simple  patterns  at  grade  4,  to  basic  algebra  concepts 
at  grade  8,  to  sophisticated  analysis  at  grade  12,  and  involves  not  only  algebra  but  also 
precalculus  and  some  topics  from  discrete  mathematics.  As  described  in  the  NCTM  Standards , 
these  algebraic  concepts  are  developed  throughout  the  grades  with  informal  modeling  done  at  the 
elementary  level  and  with  increased  emphasis  on  functions  at  the  secondary  level.  The  nature  of 
the  algebraic  concepts  and  procedures  included  in  the  assessment  at  all  levels  reflects  the  NCTM 
Standards.  Students  will  be  expected  to  use  algebraic  notation  and  thinking  in  meaningful 
contexts  to  solve  mathematical  and  real-world  problems,  specifically  addressing  an  increasing 
understanding  of  the  use  of  functions  (including  algebraic  and  geometric)  as  a representational 
tool. 
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Figure  2-2 

Descriptions  of  Mathematic  at  Abilities 


Conceptual  Understanding 

Students  demonstrate  conceptual  understanding  in  mathematics  when  they  provide 
evidence  that  they  can  recognize,  label,  and  generate  examples  and  nonexamples  of  concepts;  use 
and  interrelate  models,  diagrams,  manipulatives,  and  varied  representations  of  concepts;  identify 
and  apply  principles  (i.e.,  valid  statements  generalizing  relationships  among  concepts  in 
conditional  form);  know  and  apply  facts  and  definitions;  compare,  contrast,  and  integrate  related 
concepts  and  principles  to  extend  the  nature  of  concepts  and  principles;  recognize,  interpret,  and 
apply  the  signs,  symbols,  and  terms  used  to  represent  concepts;  or  interpret  the  assumptions  and 
relations  involving  concepts  in  mathematical  settings. 

Conceptual  understanding  reflects  a student’s  ability  to  reason  in  settings  involving  the 
careful  application  of  concept  definitions,  relations,  or  representations  of  either.  Such  an  ability 
is  reflected  by  student  performance  that  indicates  the  production  of  examples,  common  or  unique 
representations,  or  communications  indicating  the  ability  to  manipulate  central  ideas  about  the 
understanding  of  a concept  in  a variety  of  ways. 

Procedural  Knowledge 

Students  demonstrate  procedural  knowledge  in  mathematics  when  they  select  and  apply 
appropriate  procedures  correctly;  verify  or  justify  the  correctness  of  a procedure  using  concrete 
models  or  symbolic  methods;  or  extend  or  modify  procedures  to  deal  with  factors  inherent  in 
problem  settings. 

Procedural  knowledge  includes  the  various  numerical  algorithms  in  mathematics  that 
have  been  created  as  tools  to  meet  specific  needs  efficiently.  Procedural  knowledge  also 
encompasses  the  abilities  to  read  and  produce  graphs  and  tables,  execute  geometric 
constructions,  and  perform  noncomputational  skills  such  as  rounding  and  ordering.  These  latter 
activities  can  be  differentiated  from  conceptual  understanding  by  the  task  context  or  presumed 
student  background — that  is,  an  assumption  that  the  student  has  the  conceptual  understanding  of 
a representation  and  can  apply  it  as  a tool  to  create  a product  or  to  achieve  a numerical  result.  In 
these  settings,  the  assessment  question  is  how  well  the  student  executed  a procedure  or  how  well 
the  student  selected  the  appropriate  procedure  to  effect  a given  task. 

Procedural  knowledge  is  often  reflected  in  a student’s  ability  to  connect  an  algorithmic 
process  with  a given  problem  situation,  to  employ  that  algorithm  correctly,  and  to  communicate 
the  results  of  the  algorithm  in  the  context  of  the  problem  setting.  Procedural  understanding  dso 
encompasses  a student’s  ability  to  reason  through  a situation,  describing  why  a particular 
procedure  will  give  the  correct  answer  for  a problem  in  the  context  described. 
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Figure  2-2  (continued) 

Descriptions  of  Mathematical  Abilities 


Problem  Solving 

In  problem  solving,  students  are  required  to  use  their  accumulated  knowledge  of 
mathematics  in  new  situations.  Problem  solving  requires  students  to  recognize  and  formulate 
problems;  determine  the  sufficiency  and  consistency  of  data;  use  strategies,  data,  models,  and 
relevant  mathematics;  generate,  extend,  and  modify  procedures;  use  reasoning  (i.e.,  spatial, 
inductive,  deductive,  statistical,  or  proportional)  in  new  settings;  and  judge  the  reasonableness 
and  correctness  of  solutions.  Problem  solving  situations  require  students  to  connect  all  of  their 
mathematical  knowledge  of  concepts,  procedures,  reasoning,  and  communication/ 
representational  skills  in  confronting  new  situations.  As  such,  these  situations  are,  perhaps,  the 
most  accurate  measures  of  students’  proficiency  in  mathematics. 


2.5  DISTRIBUTION  OF  ASSESSMENT  ITEMS 


In  the  1996  mathematics  assessment,  a portion  of  the  items  at  one  grade  also  appeared  at 
the  other  grades.  Specifically,  for  the  grades  included  in  the  State  Assessment,  104  items 
appeared  at  grade  4 only,  54  items  appeared  at  grades  4 and  8 only,  and  129  items  appeared  at 
grade  8 only. 

Tables  2-1  and  2-2  show  the  percentages  of  each  content  strand  and  mathematical  ability 
in  the  assessment  as  specified  in  the  mathematics  framework. 

Table  2-1 

Minimum  Distribution  of  It  t ms  by  Grade  and  Content  Strand 


Content  Strand 

Grade  4 

Target  Actual1 

Grade  8 

Target  Actual1 

Number  Sense,  Properties,  and  Operations 

40%-70%2 

42% 

25%-60%2 

26% 

Measurement 

20% 

18% 

15% 

16% 

Geometry  and  Spatial  Sense 

15% 

16% 

20% 

20% 

Data  Analysis,  Statistics,  and  Probability 

10% 

12% 

15% 

19% 

Algebra  and  Functions 

15% 

12% 

25% 

19% 

'Actual  percentages  are  based  on  the  classifications  agreed  upon  by  NAEP’s  Instrument  Development  Panel. 
2For  this  category,  these  percentages  are  the  minimum  and  maximum  that  arc  acceptable,  respectively. 


Table  2-2 

Minimum  Distribution  of  Items  by  Grade  and  Mathematical  Ability 


Mathematical  Ability 

Grade  4 

Target  Actual1 

Grade  8 

Target  Actual1 

Conceptual  Understanding 

33% 

37% 

33% 

38% 

Procedural  Knowledge 

33% 

22% 

33% 

25% 

33% 

41% 

33% 

37% 

Note:  Some  items  carry  multiple  classifications. 

'Actual  percentages  are  based  on  the  classifications  agreed  upon  by  NAEP's  Instrument  Development  Panel. 


2.6  DEVELOPING  THE  COGNITIVE  ITEMS 

The  development  of  cognitive  items  began  with  careful  field  testing,  both  locally  and 
nationally,  of  grade-appropriate  questions  and  tasks  for  the  assessment.  Items  were  selected  from 
a pool  of  questions  that  were  written  by  teachers  from  across  the  country  as  well  as  by 
mathematics  assessment  specialists  on  staff  at  ETS.  The  framework  stated  that  the  assessment 
should  include  some  performance-based  questions  and  tasks  that  require  students  to  reason  and 
make  connections  within  and  across  different  content  strands  of  mathematics.  Final  selections  of 
questions  used  in  the  1996  assessment  were  approved  by  the  Mathematics  Instrument 
Development  Panel. 
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The  State  Assessment  included  constructed-response  (short  and  extended)  and  multiple- 
choice  items.  The  decision  to  use  a specific  item  type  was  based  on  a consideration  of  the  most 
appropriate  format  for  assessing  the  particular  objective.  Both  types  of  constructed-response 
items  were  designed  to  provide  an  in-depth  view  of  students*  ability  to  communicate  their 
understanding  of  important  concepts  in  mathematics.  Short  constructed-response  questions 
(scored  with  either  a 2-  or  3-level  scoring  rubric)  were  used  when  students  needed  to  respond 
briefly  in  order  to  demonstrate  full  comprehension.  Extended  constructed-response  questions 
(scored  with  a 4-  or  5-level  scoring  rubric)  were  used  when  the  task  required  more  thoughtful 
consideration  of  the  problem  and  engagement  in  more  complex  reasoning  processes.  Multiple- 
choice  items  were  used  when  a straightforward,  single  correct  answer  was  required.  Guided  by 
the  NAEP  mathematics  framework,  the  Instrument  Development  Panel  monitored  the 
development  of  all  three  types  of  items  to  assess  objectives  in  the  framework.  For  more 
information  about  item  scoring,  see  Chapter  5. 

The  State  Assessment  at  each  grade  consisted  of  13  blocks  of  items,  each  15  minutes  in 
length.  As  with  the  1992  instrument  development  effort,  a detailed  series  of  steps  was  used  to 
create  the  new  assessment  items  for  1996  that  reflected  the  objectives. 

1.  Item  specifications  and  prototype  items  were  provided  in  the  1996 
Mathematics  Framework . 

2.  The  Mathematics  Instrument  Development  Panel  provided  guidance  to 
NAEP  staff  about  how  the  objectives  could  be  measured  given  the  realistic 
constraints  of  resources  and  the  feasibility  of  measurement  technology.  The 
Panel  made  recommendations  about  priorities  for  the  assessment  and  types 
of  items  to  be  developed. 

3.  items  were  chosen  for  the  assessment  through  an  extensive  selection  process 
that  involved  the  input  of  teachers  from  across  the  country  as  well  as  the 
Mathematics  Instrument  Development  Panel. 

4.  Item  writers  from  both  inside  and  outside  ETS  were  selected  based  on  their 
knowledge  about  mathematics  education  and  experience  in  creating  items 
according  to  specifications. 

5.  The  items  were  reviewed  and  revised  by  NAEP/ETS  staff  and  external  test 
specialists. 

6.  Items  were  reviewed  by  grade-appropriate  teachers  across  the  country  for 
developmental  appropriateness. 

7.  Representatives  from  the  State  Education  Agencies  met  and  reviewed  all 
items  and  background  questionnaires  (see  Section  2.8  for  a discussion  of  the 
background  questionnaires). 

8.  Language  editing  and  sensitivity  reviews  were  conducted  according  to  ETS 
quality  control  procedures. 

l>.  Field  test  materials  were  prepared,  including  the  materials  necessary  to 
secure  clearance  by  the  Office  of  Management  and  Budget. 
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10.  The  field  test  was  conducted  in  many  states,  the  District  of  Columbia,  and 
three  territories. 

1 1 . Representatives  from  State  Education  Agencies  met  and  reviewed  the  field 
test  results. 

12.  Based  on  the  field  test  analyses,  new  items  for  the  1996  assessment  were 
revised,  modified,  and  re-edited,  where  necessary.  The  items  once  again 
underwent  ETS  sensitivity  review. 

13.  The  Mathematics  Instrument  Development  Panel  approved  the  selection  of 
items  to  include  in  the  1996  assessment. 

14.  After  a final  review  and  check  to  ensure  that  each  assessment  booklet  and 
each  block  met  the  overall  guidelines  for  the  assessment,  the  booklets  were 
typeset  and  printed.  In  total,  the  items  that  appeared  in  the  State  Assessment 
underwent  86  separate  reviews,  including  reviews  by  NAEP/ETS  staff, 
external  reviewers,  State  Education  Agency  representatives,  and  federal 
officials. 


2.7  STUDENT  ASSESSMENT  BOOKLETS 

Each  student  assessment  booklet  included  one  section  of  general  background  questions, 
one  section  of  mathematics  background  questions,  a section  of  motivation  questions,  and  three 
sections  of  mathematics  cognitive  questions.  The  assembly  of  mathematics  blocks  into  booklets 
and  their  subsequent  assignment  to  sampled  students  was  determined  by  a balanced  incomplete 
block  (BIB)  design  with  spiraled  administration.  The  overall  assessment  time  for  each  student 
was  approximately  one  hour. 

The  mathematics  cognitive  blocks  were  assigned  to  booklets  in  such  a way  that  each 
block  appeared  in  the  same  number  of  booklets  and  every  pair  of  blocks  appeared  together  in 
exactly  one  booklet.  This  is  the  balanced  part  of  the  balanced  incomplete  block  design.  It  is  an 
incomplete  block  design  because  no  booklet  contained  all  items  and  hence  there  is  incomplete 
data  for  each  assessed  student. 

The  assessment  booklets  were  then  spiraled  and  bundled.  Spiraling  involves 
interweaving  the  booklets  in  a systematic  sequence  so  that  each  booklet  appears  an  appropriate 
number  of  times  in  the  sample.  The  bundles  were  designed  so  that  each  booklet  would  appear 
equally  often  in  each  position  in  a bundle. 

The  final  step  in  the  BIB-spiraling  procedure  was  the  assigning  of  the  booklets  to  the 
assessed  students.  The  students  within  an  assessment  session  were  assigned  booklets  in  the  order 
in  which  the  booklets  were  bundled.  Thus,  students  in  an  assessment  session  received  different 
booklets,  and  only  a few  students  in  a session  received  the  same  booklet.  In  most  jurisdictions  in 
the  State  Assessment,  up  to  30  students  were  selected  from  each  school,  with  the  aim  of 
providing  an  initial  sample  size  of  approximately  3,000  public-school  students  per  jurisdiction 
per  grade,  who  responded  to  each  item.  The  nonpublic-school  samples  differed  in  size  across  the 
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jurisdictions,  with  the  number  of  schools  selected  proportional  to  the  nonpublic-school 
enrollment  within  each  jurisdiction.  Typically,  about  20  to  25  nonpublic  schools  (per  grade)  were 
included  for  each  jurisdiction. 

Table  2-3  provides  the  composition  of  each  block  of  items  administered  in  the 
State  Assessment  program  in  mathematics.  Table  2-4  shows  the  order  of  the  blocks  in  each 
booklet  and  how  the  13  cognitive  blocks  were  arranged  across  the  26  booklets  to  achieve  the 
BIB-spiral  design.  The  1996  design  was  identical  to  that  used  in  1992.  The  new  blocks  that  were 
developed  for  the  1996  State  Assessment  (M5,  M7,  Ml 2,  M 14,  and  M15)  were  arranged  within 
the  booklet  design  in  the  same  manner  as  were  the  1992  blocks  that  they  replaced. 


Table  2-3 

Cognitive  and  Noncognitive  Block  Information 
Grade  4 /Grade  <S* 


Block 

Type 

Total 
Number 
of  Items 

Number  of 
Multiple- 
Choice 
Items 

Number  of 
Constructed- 
Response 
Items 

Numbers  of 
Booklets 
Containing 
Block2 

BMI 

Common  Background 

24/26 

24/26 

00/00 

26 

MB! 

Mathematics  Background 

25/3 1 

25/3 1 

00/00 

26 

MB 

Mathematics  Motivation 

05/05 

05/05 

00/00 

26 

M3 

Mathematics  Cognitive  (Trend) 

13/13 

09/09 

04/04 

6-7 

M4 

Mathematics  Cognitive1  (Trend) 

14/21 

14/21 

00/00 

6-7 

M5 

Mathematics  Cognitive1 

10/1  1 

04/06 

06/05 

6-7 

M6 

Mathematics  Cognitive*  (Trend) 

1 1/16 

00/00 

1 1/16 

6-7 

M7 

Mathematics  Cognitive1  (Manipulate ex) 

0X/1 0 

03/05 

05/05 

6-7 

MS 

Mathematics  Cognitive  (Calculator) 

1 5/1 X 

14/16 

01/02 

6-7 

M9 

Mathematics  Cognitive 

1 2/09 

09/05 

03/04 

6-7 

M 10 

Mathematics  Cognitive1  (Munipulatives) 

06/07 

00/00 

06/07 

6-7 

Ml  1 

Mathematics  Cognitive1 

16/19 

1 1/13 

05/06 

6-7 

MI  2 

Mathematics  Cognitive  (Calculator) 

09/09 

05/04 

04/05 

6-7 

Ml  3 

Mathematics  Cognitive 

12/1 1 

06/06 

06/05 

6-7 

M 14 

Mathematics  Cognitive  (Calculator) 

10/00 

04/05 

06/04 

6-7 

M 1 5 

Mathematics  Cognitive 
(Ruler/Protractor) 

10/09 

03/04 

07/05 

6-7 

' These  blocks  contained  some  items  that  appeared  at  both  grades  4 and  X 

'Booklets  that  contained  a *0  minute  theme  block  had  six  sections;  booklets  that  did  not  contain  a W-minutc  theme 
block  had  seven  sections  There  wete  no  theme  block  booklets  in  the  !Wi  State  Assessment. 
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Table  2-4 

Booklet  Contents  for  Both  Grades 


Booklet 

Number 

Common 

Background 

Block 

Mathematics 

Background 

Block 

Cognitive  Blocks  Mathematics 

, Motivation 

1st  2nd  3rd  Block 

M101 

BM1 

MBl 

M3 

M4 

M7 

MB 

Ml  02 

BM1 

MB1 

M4 

M5 

M8 

MB 

Ml  03 

BM1 

MBl 

M5 

M6 

M9 

MB 

Ml  04 

BMI 

MBl 

M6 

M7 

M10 

MB 

Ml  05 

BM1 

MBl 

M7 

M8 

Mil 

MB 

Ml  06 

BMI 

MBl 

M8 

M9 

M12 

MB 

M 107 

BMI 

MBl 

M9 

M 10 

M13 

MB 

Ml  08 

BMI 

MBl 

M 10 

Mil 

M14 

MB 

Ml  09 

BMI 

MBl 

Mil 

M12 

M15 

MB 

MHO 

BMI 

MBl 

M12 

M13 

M3 

MB 

Mill 

BMI 

MBl 

M13 

M14 

M4 

MB 

Ml  12 

BMI 

MBl 

M14 

M 15 

M5 

MB 

Ml  13 

BMI 

MBl 

M15 

M3 

M6 

MB 

Ml  14 

BMI 

MBl 

M3 

M5 

M10 

MB 

Ml  15 

BMI 

MBl 

M4 

M6 

Mil 

MB 

Ml  16 

BMI 

MBl 

M5 

M7 

Ml  2 

MB 

Ml  17 

BMI 

MBl 

M6 

M8 

Ml  3 

MB 

Ml  18 

BMI 

MBl 

M7 

M9 

M14 

MB 

Ml  19 

BMI 

MBl 

M8 

M10 

M15 

MB 

Ml  20 

BMI 

MBl 

M9 

Mil 

M3 

MB 

Ml  21 

BMI 

MBl 

M 10 

M12 

M4 

MB 

Ml  22 

BMI 

MBl 

Mil 

Ml  3 

M5 

MB 

M 1 23 

BMI 

MBl 

M12 

M14 

M6 

MB 

Ml  24 

BMI 

MBl 

M13 

M15 

M7 

MB 

Ml  25 

BMI 

MBl 

M 14 

M3 

M8 

MB 

Ml  26 

BMI 

MBl 

M15 

M4 

M9 

MB 

2.8  QUESTIONNAIRES 

As  part  of  the  State  Assessment  (as  well  as  the  national  assessment),  a series  of 
questionnaires  was  administered  to  students,  teachers,  and  school  administrators.  Similar  to  the 
development  of  the  cognitive  items,  the  development  of  the  policy  issues  and  questionnaire  items 
was  a consensual  process  that  involved  staff  work,  field  testing,  and  review  by  external  advisory 
groups.  A Background  Questionnaire  Panel  drafted  a set  of  policy  issues  and  made 
recommendations  regarding  the  design  of  the  questions.  They  were  particularly  interested  in 
capitalizing  on  the  unique  properties  of  NAEP  and  not  duplicating  other  surveys  (e.g„  the 
National  Survey  of  Public  and  Private  School  Teachers  and  Administrators,  the  School  and 
Staffing  Survey,  and  the  National  Educational  Longitudinal  Study). 
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The  Panel  recommended  a focused  study  that  addressed  the  relationship  between  student 
achievement  and  instructional  practices.  The  policy  issues,  items,  and  field  test  results  were 
reviewed  by  the  group  of  external  consultants  who  identified  specific  items  to  be  included  in  the 
final  questionnaires.  In  addition,  the  Mathematics  Instrument  Development  Panel  and  state 
representatives  were  consulted  on  the  appropriateness  of  issues  addressed  in  the  questionnaires 
as  they  relate  to  mathematics  instruction  and  achievement.  The  items  underwent  internal  ETS 
review  procedures  to  ensure  fairness  and  quality  and  were  then  assembled  into  questionnaires. 


2.8.1  Student  Questionnaires 

In  addition  to  the  cognitive  items,  the  1996  State  Assessment  included  three  student 
questionnaires:  two  sets  of  general  and  mathematics  background  questions  designed  to  gather 
contextual  information  about  students,  their  instructional  experiences  in  mathematics,  and  their 
attitudes  toward  mathematics,  and  one  set  of  background  questions,  given  to  students  at  the  end 
of  each  booklet  to  determine  their  motivation  in  completing  the  assessment  and  their  familiarity 
with  assessment  tasks.  In  order  to  ensure  that  all  fourth-grade  students  understood  the  questions 
and  had  every  opportunity  to  respond  to  them,  the  three  questionnaires  were  read  aloud  by 
administrators  as  fourth-grade  students  read  along  and  responded  in  their  booklets.  Background 
questionnaires  were  not  read  aloud  to  eighth-grade  students. 

The  student  demographics  (common  background)  questionnaire  included  questions 
about  race/ethnicity,  language  spoken  in  the  home,  mother’s  and  father’s  level  of  education, 
reading  materials  in  the  home,  homework,  attendance,  which  parents  live  at  home,  and  which 
parents  work.  This  questionnaire  was  the  first  section  in  every  booklet.  In  many  cases  the 
questions  used  were  continued  from  prior  assessments,  so  as  to  document  changes  in  contextual 
factors  that  occur  over  time. 

The  second  section  of  background  questions  was  the  mathematics  background 
questionnaire.  Categories  of  information  represented  in  this  section  include: 

Time  Spent  Studying  Mathematics:  Students  were  asked  to  describe  both  the 
amount  of  instruction  they  received  in  mathematics  and  the  time  spent  on 
mathematics  homework. 

Instructional  Practices:  Students  were  asked  to  report  their  instructional 
experiences  related  to  mathematics  in  the  classroom,  including  group  work,  special 
projects,  and  writing  in  response  to  mathematics.  In  addition,  they  were  asked 
about  the  instructional  practices  of  their  mathematics  teachers  and  the  extent  to 
which  the  students  themselves  discussed  what  they  did  in  class  and  demonstrated 
use  of  skills  and  strategies. 

Attitudes  Towards  Mathematics:  Students  were  asked  a series  of  questions  about 
their  attitudes  and  perceptions  about  mathematics. 

The  student  motivation  questionnaire  asked  students  to  describe  how  hard  they  tried 
on  the  NAEP  mathematics  assessment,  how  difficult  they  found  the  assessment,  how  many 
questions  they  thought  they  got  right,  how  important  it  was  for  them  to  do  well,  and  how  familiar 
they  were  with  the  assessment  format. 
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2.8.2  Teacher,  School,  and  SD/LEP  Student  Questionnaires 


To  supplement  the  information  on  instruction  reported  by  students,  the  mathematics 
teachers  of  the  students  participating  in  the  State  Assessment  were  asked  to  complete  a 
questionnaire  about  their  instructional  practices,  teaching  backgrounds,  and  characteristics.  The 
teacher  questionnaire  contained  two  parts.  The  first  part  pertained  to  the  teachers’  background 
and  general  training.  The  second  part  pertained  to  specific  training  in  teaching  mathematics  and 
the  procedures  the  teacher  uses  for  each  class  containing  an  assessed  student,  as  well  as 
collecting  information  on  teachers’  awareness  and  knowledge  of  the  NCTM  Standards. 

The  Teacher  Questionnaire,  Part  I:  Background  and  General  Training  included 
questions  pertaining  to  gender,  race/ethnicity,  years  of  teaching  experience,  certification, 
degrees,  major  and  minor  fields  of  study,  course  work  in  education,  course  work  in  specific 
subject  areas,  amount  of  in-service  training,  extent  of  control  over  instructional  issues,  and 
availability  of  resources  for  their  classroom. 

The  Teacher  Questionnaire,  Part  II:  Training  in  Mathematics  and  Classroom 
Instructional  Information  included  questions  on  the  teacher’s  exposure  to  various  issues 
related  to  mathematics  and  teaching  mathematics  through  pre-  and  in-service  training,  ability 
level  of  students  in  the  class,  whether  students  were  assigned  to  the  class  by  ability  level,  time  on 
task,  homework  assignments,  frequency  of  instructional  activities  used  in  class,  methods  of 
assessing  student  progress  in  mathematics,  instructional  emphasis  given  to  the  mathematics 
abilities  covered  in  the  assessment,  and  use  of  particular  resources. 

A School  Characteristics  and  Policies  Questionnaire  was  given  to  the  principal  or 
other  administrator  of  each  school  that  participated  in  the  State  Assessment  program.  This 
information  provided  an  even  broader  picture  of  the  instructional  context  for  students’ 
mathematics  achievement.  This  questionnaire  included  questions  about  background  and 
characteristics  of  school  principals,  length  of  school  day  and  year,  school  enrollment, 
absenteeism,  drop-out  rates,  size  and  composition  ot  teaching  staff,  policies  about  grouping 
students,  curriculum,  testing  practices  and  uses,  special  priorities  and  school-wide  programs, 
availability  of  resources,  special  services,  community  services,  policies  for  parental  involvement, 
and  school-wide  problems. 

The  SD/LEP  Student  Questionnaire  was  completed  by  the  teachers  of  students  who 
were  selected  to  participate  in  the  State  Assessment  sample  who  were  also  identified  as  students 
with  a disability  (SD)  or  categorized  as  being  of  limited  English  proficiency  (LEP).  Some  of 
these  students  were  determined  by  the  school  to  be  ineligible  to  be  assessed.  In  order  to  be 
excluded  from  the  assessment,  a student  must  have  been  identified  as  SD  and  must  not  have  been 
mainstreamed  at  least  50  percent  of  the  time,  or  was  categorized  as  LEP.  In  addition,  the  school 
staff  would  have  needed  to  determine  that  it  was  inappropriate  to  include  the  student  in  the 
assessment.  This  questionnaire  asked  about  the  nature  of  the  student’s  disability  or  about  the 
student's  native  language,  and  the  special  programs  in  which  the  student  participated. 


2.9  DEVELOPMENT  OF  FINAL  FORMS 

The  field  tests  of  new  items  for  the  1996  assessment  were  conducted  in  February  and 
March  1993  and  involved  students  in  many  states,  the  District  of  Columbia,  and  three  U.  S. 
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territories.  The  intent  of  the  field  test  was  to  try  out  the  items  and  procedures  and  to  give  the 
states  and  the  contractors  practice  and  experience  with  the  proposed  materials  and  procedures. 
About  500  responses  were  obtained  to  each  item  in  the  field  test. 

The  field  test  data  were  collected,  scored,  and  analyzed  in  preparation  for  meetings  with 
the  Mathematics  Instrument  Development  Panel.  The  objectives  that  guided  the  review  of  these 
items  were: 


• to  determine  which  items  were  most  suitable  for  assessing  understanding  in 
mathematics  in  accordance  with  the  framework; 

• to  determine  the  need  for  revisions  of  items  that  lacked  clarity,  or  had 
ineffective  item  formats; 

• to  prioritize  items  to  be  included  in  the  State  Assessment;  and, 

• to  determine  appropriate  timing  for  assessment  items. 

Committee  members,  ETS  assessment  staff,  and  NAEP/ETS  staff  reviewed  the  materials. 
Item  analyses  (which  provided  the  mean  percentage  of  correct  responses,  the  r-biserial 
correlations,  and  the  difficulty  level  for  each  item)  were  used  as  a guide  in  identifying  and 
flagging  for  further  review  those  test  items  that  were  not  measuring  the  intended  objective  well. 
In  addition,  another  meeting  of  representatives  from  state  education  agencies  was  convened  to 
review  the  field  test  results. 

Once  the  committees  had  selected  the  items,  all  items  were  rechecked  for  content, 
measurement,  and  sensitivity  concerns.  The  federal  clearance  process  was  initiated  in  June  1993 
with  the  submission  of  draft  materials  to  NCES.  The  final  package  containing  the  final  set  of 
cognitive  items  assembled  into  blocks  and  questionnaires  was  submitted  in  August  1993. 
Throughout  the  clearance  process,  revisions  were  made  in  accordance  with  changes  required  by 
the  government.  After  approval,  the  blocks  (assembled  into  booklets)  and  questionnaires  were 
readied  for  printing  in  preparation  for  the  assessment. 


Chapter  3 


SAMPLE  DESIGN  AND  SELECTION  1 

John  Burke  and  James  L.  Green 
Westat,  Inc. 


3.1  OVERVIEW 

The  1996  State  Assessment  program  in  mathematics  included  assessments  of  fourth-  and 
eighth-grade  students  in  public-  and  nonpublic-schools.  The  1996  State  Assessment  program  in 
science  was  conducted  at  grade  8 only,  except  for  Department  of  Defense  Education  Activity 
(DoDEA)  schools  that  were  assessed  at  both  grades  4 and  8.  For  the  eighth-grade,  the  samples 
selected  for  both  the  mathematics  and  science  assessment  were  selected  as  part  of  the  same 
process.  Some  schools  that  were  selected  for  participation  in  the  eighth-grade  sample  provided 
both  students  that  were  assessed  in  mathematics  and  students  that  were  assessed  in  science.  (This 
was  also  true  for  the  DDESS  and  DoDDS  sample.)  A representative  sample  of  public-  and 
nonpublic-school  students  was  drawn  in  each  participating  jurisdiction.  Each  sample  was 
designed  to  produce  aggregate  estimates  as  well  as  estimates  for  various  subpopulations  of 
interest  with  approximately  equal  precision  for  the  participating  jurisdictions.  The  sample  for  the 
fourth-  and  eighth-grade  public-school  assessments  in  each  jurisdiction  consisted  of  about  3,150 
students  (before  attrition)  in  each  subject  from  about  1 00  public  schools  in  each  case.  The  target 
for  nonpublic-school  students  varied  by  jurisdiction  and  was  proportional  to  their  representation 
in  the  jurisdiction. 

The  target  population  for  the  1996  State  Assessment  program  included  students  in  public 
and  nonpublic  schools  who  were  enrolled  in  the  fourth  and  eighth  grade  at  the  time  of 
assessment.  The  sampling  frame  included  public  and  nonpublic  schools  having  the  relevant  grade 
in  each  jurisdiction.  The  samples  were  selected  based  on  a two-stage  sample  design;  selection  of 
schools  within  participating  jurisdictions,  and  selection  of  students  within  schools.  The 
first-stage  samples  of  schools  were  selected  with  probability  proportional  to  a measure  of  size 
based  on  the  estimated  grade-specific  enrollment  in  the  schools.  Special  procedures  were  used 
for  jurisdictions  with  many  small  schools,  and  for  jurisdictions  having  small  numbers  of  grade- 
eligible  schools. 

Stratification  variables  were  added  to  the  sampling  frame  prior  to  sample  selection. 
Public  schools  were  stratified  by  urbanization  and  minority  class  and  nonpublic  schools  were 
stratified  by  metro  area  status  and  school  type.  The  urbanization  strata  were  defined  in  terms  of 
large  or  mid-size  central  city,  urban  fringe  of  large  or  mid-size  city,  large  town,  small  town,  and 
rural  areas.  Within  urbanization  strata,  public  schools  were  further  stratified  explicitly  on  the 
basis  of  minority  enrollment  in  those  jurisdictions  with  substantial  Black  or  Hispanic  student 
population.  Minority  enrollment  was  defined  as  the  total  percent  of  Black  and  Hispanic  students 
enrolled  in  a school.  Within  minority  strata,  public  schools  were  sorted  by  median  household 
income  of  the  ZIP  code  area  where  the  school  was  located.  Metro  area  status  was  determined  by 
U.S.  Bureau  of  Census  definitions  as  of  June  30,  1993.  Other  stratification  variables  were 
obtained  from  Quality  Education  Data,  Inc.  (QED)  and  the  National  Center  for  Education 


1 John  Burke  was  responsible  for  overseeing  all  sampling  activities;  James  Green  earned  out  most  of  these  activities 
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Statistics  Common  Core  of  Data  (CCD).  School  type  was  a dichotomous  variable  (Catholic  or 
other  nonpublic).  Within  school  type,  nonpublic  schools  were  sorted  by  estimated  grade 
enrollment. 

From  the  stratified  frame  of  public  and  nonpublic  schools  within  each  jurisdiction,  a 
systematic  random  sample  of  about  100  grade-eligible  schools  was  drawn  with  probability 
proportional  to  a measure  of  size  based  on  the  estimated  grade-specific  enrollment  of  the  school. 
One  or  more  sessions  were  sampled  within  each  school.  The  number  of  sessions  selected 
depended  on  the  school’s  estimated  grade-specific  enrollment,  though  the  overwhelming 
majority  of  schools  at  grade  4 were  allocated  a single  session. 

For  jurisdictions  that  participated  in  the  1994  Trial  State  Assessment,  25  percent  of  the 
selected  public  and  nonpublic  schools  were  designated  at  random  to  be  monitored  during  the 
assessment  field  period  so  that  reliable  comparisons  could  be  made  between  sessions 
administered  with  and  without  monitoring.  For  jurisdictions  that  did  not  participate  in  the 
previous  assessment," 50  percent  of  the  selected  public  and  nonpublic  schools  were  designated  to 
be  monitored. 

Approximately  3,150  public-school  students  were  targeted  for  selection  for  a given  grade 
and  subject  in  a given  jurisdiction.  For  nonpublic  schools,  the  target  for  each  grade  and  subject 
varied  by  jurisdiction.  On  average,  109  public  schools  and  20  nonpublic  schools  were  selected 
for  fourth  grade  in  each  jurisdiction  and  105  public  schools  and  31  nonpublic  schools  were 
selected  for  eighth  grade  in  each  jurisdiction.  The  maximum  number  of  public  and  nonpublic 
schools  sampled  in  a jurisdiction  were  139  and  44,  respectively,  for  fourth  grade.  The  minimum 
number  of  public  and  nonpublic  schools  sampled  in  a jurisdiction  were  22  and  10,  respectively, 
for  fourth  grade.  The  maximum  number  of  public  and  nonpublic  schools  sampled  for  eighth 
grade  were  159  and  68,  respectively.  The  minimum  number  of  public  and  nonpublic  schools 
sampled  in  a jurisdiction  were  6 and  10,  respectively,  for  eighth  grade. 

Each  selected  school  provided  a list  of  eligible  enrolled  students,  from  which  a 
systematic  sample  of  students  was  drawn.  Thirty  students  were  selected  for  each  session. 

The  1996  State  Assessment  was  preceded  in  1995  by  a field  test.  The  principal  goals  of 
the  field  test  were:  ( 1)  to  test  new  items  contemplated  for  1996,  and  (2)  to  test  procedures 
contemplated  for  1996.  Some  schools  that  participated  in  the  Field  test  were  also  given  a chance 
of  selection  in  the  1996  assessment.  Section  3.2  documents  the  procedures  used  to  select  the 
schools  for  the  field  test. 

The  rest  of  this  chapter  documents  the  procedures  used  to  select  schools  for  the  1996 
State  Assessment.  Section  3.3  describes  the  construction  of  the  sampling  frames,  including  the 
sources  of  school  data,  missing  data  problems,  and  definition  of  in-scope  schools.  Section  3.4 
includes  a description  of  the  various  steps  in  stratification  of  schools  within  participating 
jurisdictions.  School  sample  selection  procedures  (including  new  and  substitute  schools)  are 
described  in  Section  3.5.  Section  3.5.5  includes  information  about  the  selection  of  schools  for 
application  of  the  two  sets  of  inclusion  rules  (SI  and  S2  subsamples)  used  in  the  State 
Assessment.  Section  3.6  includes  the  steps  involved  in  selection  of  students  within  participating 
schools. 
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3.2  SAMPLE  SELECTION  FOR  THE  1995  FIELD  TEST 

The  1995  field  test  for  the  State  Assessment  program  was  conducted  together  with  the 
field  test  for  the  national  portion  of  the  assessment.  In  these  field  tests,  assessments  were  piloted 
in:  mathematics,  science,  and  the  arts  (dance,  music,  theater,  and  visual  arts).  All  jurisdictions 
were  included  in  the  field  test  except  Alaska,  Delaware,  the  District  of  Columbia,  Hawaii,  Rhode 
Island  and  Wyoming,  which  were  excluded  due  to  the  heavy  burden  placed  on  these  small 
population  jurisdiction:  -y  the  main  assessment.  The  field  test  was  conducted  for  grades  4,  8, 
and  12.  Pairs  of  schools  were  identified,  with  one  of  each  pair  to  be  included  in  the  test.  This 
allowed  state  participation  in  the  selection  of  the  test  schools  and  also  facilitated  replacement  ot 
schools  that  declined  to  participate  in  the  assessment.  Sampling  weights  were  not  computed  for 
the  field  test  samples. 


3.2.1  Primary  Sampling  Units 

The  field  test  primary  sampling  unit  (PSU)  sampling  frame  was  derived  from  the  national 
list  of  U.S.  counties.  The  frame  was  stratified  by  state  and  metro  area  status.  Two  hundred  and 
fifteen  PSUs  were  selected  from  the  resulting  field  test  frame.  Twenty  PSUs  were  selected  with 
certainty  and  195  noncertainty  PSUs  were  selected— one  per  noncertainty  stratum.  The  PSUs 
were  selected  systematically  and  with  probability  proportional  to  the  1990  PSU  population. 
Counties  that  were  noncertainty  selections  for  the  Third  International  Mathematics  and  Science 
Study  (TIMSS),  the  1996  NAEP  national  assessment,  and  the  1996  NAEP  trend  samples  were 
excluded  from  the  sampling  frame.  The  number  of  counties,  and  so,  PSU  s,  selected  per 
jurisdiction  ranged  from  2 to  10. 


3.2.2  Selection  of  Schools  and  Students 

Public  and  nonpublic  schools  with  fourth-,  eighth-,  or  twelfth-grade  students  were 
in-scope  for  the  field  test  assessment.  Schools  with  fewer  than  40  students  were  eliminated  from 
the  sampling  frame  to  avoid  the  relatively  high  per  student  cost  of  conducting  assessments  in 
small  schools.  Schools  selected  as  originals  or  substitutes  for  TIMSS  were  also  eliminated  from 

the  frame. 

Across  all  three  grades  from  the  resulting  sampling  frame,  1.285  pairs  of  schools  were 
selected  The  first  member  of  each  pair  was  selected  systematically  and  with  probability 
proportional  to  grade  enrollment.  The  twelfth-grade  sample  was  drawn  first  followed  by  the 
eighth-  and  fourth-  grade  samples.  The  selected  twelfth-grade  schools  were  removed  from  the 
frame  before  drawing  the  eighth-grade  sample.  The  selected  twelfth-  and  eighth-grade  schoo  s 
were  removed  from  the  frame  before  drawing  the  fourth-grade  sanmK  In  this  way,  no  school 
was  selected  for  more  than  one  grade. 

The  second  member  of  each  pair  was  selected  in  such  a way  that  the  “distance  from  the 
primary  selection,  based  on  percent  of  Black  students,  percent  of  Hispanic  students,  grade 
enrollment,  and  percent  of  students  living  below  poverty  was  the  smallest  across  all  schools 
remaining  after  the  fourth-,  eighth-,  and  twelfth-grade  sampling. 
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3.2.3  Assignment  to  Sessions  for  Different  Subjects 

Up  to  six  different  session  types  were  assigned  in  a given  jurisdiction.  The  particular 
number  of  session  types  varied  by  grade.  Table  3- 1 gives  the  overall  number  of  schools  selected 
for  each  grade  and  session  type.  The  number  of  sessions  assigned  to  an  individual  school 
depended  on  the  size  of  the  school  and  the  subject(s)  that  school  was  assigned. 


Table  3-1 

Number  of  Schools  Selected  for  Each  Grade  and  Session  Type 


Session  Type1 

Grade  4 

Grade  8 

Grade  12 

Mathematics/Science 

75 

80 

90 

Mathematics  Trend 

75 

75 

75 

Visual  Arts/Music 

15 

70 

100 

Theater  and  Dance 

85 

85 

120 

Spanish/Bilingual 

120 

120 

0 

SD  Accommodations 

20 

20 

0 

Total 

450 

450 

385 

The  mathematics  and  science  sessions  were  sessions  where  items  selected  for  the  1996  State  Assessment  program  were 
administered.  The  mathematics  trend  sessions  were  sessions  where  booklets  from  the  1996  Trial  State  Assessment 
program  in  mathematics  were  administered.  The  results  from  students  included  in  these  sessions  were  used  to  verify 
that  the  1996  and  1992  assessments  could  be  placed  on  the  same  scale.  The  two  types  of  arts  sessions  were  selected  for 
every  grade.  They  were  administered  at  grades  4 and  8,  although  the  grade  12  sessions  were  administered  the  arts  field 
test  in  1997.  The  Spanish/bilingual  and  SD  accommodations  sessions  were  administered  using  special  booklets  to 
determine  whether  SD/LEP  students  could  participate  in  the  1996  assessment  with  special  accommodations. 


3.3  TARGET  POPULATION  AND  SAMPLING  FRAME  FOR  THE  1996 
ASSESSMENT 

3.3.1  Target  Population 

The  target  population  for  the  1996  State  Assessment  included  students  in  public  and 
nonpublic  schools  who  were  enrolled  in  the  fourth  or  eighth  grade.  Nonpublic  schools  included 
Catholic  schools,  other  religious  schools,  private  schools,  DDESS,  and  Bureau  of  Indian  Affairs 
(BIA)  schools.  Special  education  schools  were  not  included.  Both  SI,  based  on  the  old  inclusion 
rules,  and  S2,  based  on  the  new  inclusion  rules,  shared  this  target  population. 


3.3.2  Sampling  Frame 


In  order  to  draw  the  school  samples  for  the  1996  State  Assessment,  it  was  necessary  to 
obtain  a comprehensive  list  of  public  and  nonpublic  schools  in  each  jurisdiction.  For  each  school, 
useful  information  for  stratification  purposes,  reliable  information  about  grade  span  and 
enrollment,  and  accurate  information  for  identifying  the  school  to  the  state  coordinator  (district 
membership,  name,  address)  were  required. 
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Based  on  the  experience  with  the  1992  and  1994  Trial  State  Assessments,  and  national 
assessments  from  1984  to  1994,  the  file  made  available,  by  QED  was  elected  as  the  sampling 
frame.  The  CCD  school  file  was  used  to  check  the  completeness  of  the  QED  file. 

The  QED  list  covers  all  U.S.  states  and  jurisdictions  except  Puerto  Rico.  The  version  of 
the  QED  file  used  was  released  in  late  1994,  in  time  for  selection  of  the  school  sample  in  early 
1995.  The  file  was  missing  racial/ethnic  minority  enrollment  and  urbanization  data  for  a sizable 
minority  of  schools  (due  to  the  inability  of  QED  to  match  these  schools  with  the  corresponding 
CCD  file).  Considerable  efforts  were  undertaken  to  obtain  these  variables  for  all  schools  in 
jurisdictions  where  these  variables  were  to  be  used  for  stratification.  These  efforts  are  described 
in  the  next  section. 

A new  addition  for  1996  was  the  joint  use  of  QED  and  National  Center  for  Education 
Statistics’  Private  School  Universe  Survey  (PSS)  lists  of  nonpublic  schools.  These  two  sources 
were  combined,  eliminating  duplicates  as  necessary  and  increasing  coverage  throughout  the 
combined  frame.  When  a given  school  was  found  on  both  lists,  the  PSS  data  were  given  priority. 

Tables  3-2  and  3-3  show  the  distribution  of  fourth-  and  eighth-grade  schools  and 
enrollment  within  schools  as  reported  in  the  combined  frame.  Grade-specific  enrollment  was 
estimated  for  each  school  as  the  quotient  of  total  school  enrollment  and  the  number  of  grades  in 
the  school. 
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Table  3-2 

Distribution  of  Fourth-Grade  Schools  and  Enrollment  in  Combined  Frame 


Public  Schools  Nonpublic  Schools 


Jurisdiction 

Total  Schools 

Total  Enrollment 

Total  Schools 

Total  Enrollment 

Alabama 

760 

58,446 

271 

6,319 

Alaska 

344 

9,838 

70 

650 

Arizona 

685 

59,386 

265 

4,772 

Arkansas 

535 

35,580 

152 

2,566 

California 

4,823 

436,578 

2,521 

57,037 

Colorado 

764 

51,068 

251 

4,517 

Connecticut 

565 

40,703 

248 

5,497 

Delaware 

52 

7,573 

108 

2,200 

District  of  Columbia 

107 

6,156 

53 

1,602 

DoDEA/DDESS 

39 

3,118 

N/A 

N/A 

DoDEA/DoDDS 

113 

7,291 

N/A 

N/A 

Florida 

1,444 

167,900 

1,019 

23,748 

Georgia 

1,034 

104,129 

455 

10,162 

Guam 

21 

2,356 

14 

566 

Hawaii 

176 

14,848 

100 

2,857 

Indiana 

1,130 

74,784 

640 

10,537 

Iowa 

758 

38,645 

236 

5,278 

Kentucky 

796 

50,603 

272 

6,851 

Louisiana 

793 

63,887 

369 

12,201 

Maine 

394 

17,589 

109 

1,254 

Maryland 

792 

60,409 

426 

10,638 

Massachusetts 

1,030 

72,171 

421 

10,191 

Michigan 

1,871 

129,209 

943 

18,573 

Minnesota 

827 

63,255 

481 

9,013 

Mississippi 

458 

41,251 

171 

4,580 

Missouri 

1,093 

67,929 

492 

11,110 

Montana 

458 

13,482 

88 

1,076 

Nebraska 

882 

22,310 

202 

3,786 

Nevada 

244 

18,780 

52 

974 

New  Jersey 

1,315 

91,263 

698 

18,578 

New  Mexico 

384 

26,072 

187 

2,668 

New  York 

2,255 

204,385 

1,526 

43,627 

North  Carolina 

1,116 

91,917 

441 

8,536 

r 
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Table  3-2  (continued) 

Distribution  of  Fourth-Grade  Schools  and  Enrollment  in  Combined  Frame 


Jurisdiction 

Public  Schools 

Total  Schools  Total  Enrollment 

Nonpublic  Schools 

Total  Schools  Total  Enrollment 

North  Dakota 

325 

9,704 

77 

1,002 

Oregon 

752 

41,320 

234 

3,602 

Pennsylvania 

1,831 

136,418 

1,558 

31,902 

Rhode  Island 

178 

11,688 

78 

1,926 

South  Carolina 

552 

51,328 

251 

5,062 

Tennessee 

929 

70,073 

342 

6,825 

Texas 

3,181 

285,699 

931 

21,560 

Utah 

434 

37,425 

55 

959 

Vermont 

247 

8,051 

58 

676 

Virginia 

1,042 

84,306 

409 

7,788 

Washington 

1,051 

74,964 

406 

7,275 

West  Virginia 

553 

23,628 

137 

1,322 

Wisconsin 

1,137 

65,421 

873 

14,968 

Wyoming 

226 

8,032 

35 

281 

Total 

40,323 

3,048,203 

18,711 

406,546 

Table  33 

Distribution  of  Eighth-Grade  Schools  and  Enrollment  In  Combined  Frame 

Public  Schools 

Nonpublic  Schools 

Jurisdiction 

Total  Schools 

Total  Enrollment 

Total  Schools 

Total  Enrollment 

Alabama 

484 

56,995 

245 

5,363 

Alaska 

256 

9,240 

59 

481 

Arizona 

328 

54,351 

227 

4,210 

Arkansas 

344 

35,074 

110 

1,846 

California 

1,642 

379,030 

2,023 

47,939 

Colorado 

325 

46,695 

224 

3,795 

Connecticut 

207 

34,383 

248 

5,828 

Delaware 

29 

7,751 

101 

2,097 

Pistrict  of  Columbia  32 

4,808 

46 

1,435 

DoDEA/DDESS 

12 

1,517 

N/A 

N/A 

DoDEA/DoDDS 

66 

5,353 

N/A 

N/A 

Florida 

466 

152,838 

839 

19,767 

Georgia 

398 

97,029 

385 

8,297 

Guam 

6 

2,199 

12 

498 

Hawaii 

52 

12.845 

84 

3,341 

Indiana 

437 

76,101 

558 

9.073 

Iowa 

409 

38,331 

194 

4,461 

Kentucky 

357 

51,275 

238 

6,293 

Louisiana 

431 

59,102 

352 

13,767 

Maine 

235 

16,134 

98 

1,077 

Maryland 

229 

57,586 

CC 
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Table  3-3  (continued) 

Distribution  of  Eighth-Grade  Schools  and  Enrollment  In  Combined  Frame 


Public  Schools  Nonpublic  Schools 


Jurisdiction 

Total  Schools 

Total  Enrollment 

Total  Schools 

Total  Enrollment 

Massachusetts 

383 

61,789 

407 

10,656 

Michigan 

737 

120,422 

819 

16,577 

Minnesota 

424 

59,224 

361 

7,447 

Mississippi 

301 

39,570 

143 

4,076 

Missouri 

633 

63,768 

441 

10,375 

Montana 

321 

12,800 

81 

834 

Nebraska 

577 

22,137 

173 

3,502 

Nevada 

95 

18,626 

44 

840 

New  Hampshire 

132 

14,600 

78 

1,228 

New  Jersey 

664 

84,346 

660 

18,516 

New  Mexico 

152 

24,249 

148 

2,387 

New  York 

1,013 

187,305 

1,368 

42,412 

North  Carolina 

526 

89,074 

377 

6,856 

North  Dakota 

237 

9,065 

54 

743 

Oregon 

343 

39,630 

195 

2,808 

Rhode  Island 

52 

10,286 

77 

2,163 

South  Carolina 

252 

51,010 

206 

3,679 

Tennessee 

533 

66,684 

325 

7,044 

Texas 

1,488 

271,798 

680 

16,095 

Utah 

142 

36,877 

54 

913 

Vermont 

126 

7,413 

52 

575 

Virginia 

336 

79,009 

362 

7,124 

Washington 

425 

70,998 

345 

6,430 

West  Virginia 

206 

24,448 

126 

1,214 

Wisconsin 

513 

61,628 

778 

13,729 

Wyoming 

96 

7,971 

31 

195 

Total 

21,740 

3,243,013 

18,452 

423,591 

3.4  STRATIFICATION 
3.4.1  Stratification  Variables 

The  stratification  used  for  sample  selection  varied  by  school  type  (public  or  nonpublic). 
Stratification  of  public  schools  involved  four  primary  dimensions  whereas  the  stratification  of 
nonpublic  schools  involved  three  primary  dimensions.  Public  schools  were  stratified 
hierarchically  by  small  or  large  district  status,  school  size  class,  urbanization  classification  and 
minority  classification.  Nonpublic  schools  were  stratified  by  size  class,  metro  area  status  and 
school  type  (Catholic  or  other  nonpublic).  Public  schools  were  further  stratified  implicitly  by 
median  household  income  (i.e.,  sorted  in  ascending  or  descending  order)  of  the  ZIP  code  area 
where  the  school  was  located,  and  nonpublic  schools  were  further  stratified  implicitly  by 
estimated  grade  enrollment  in  order  to  provide  some  control  of  these  variables.  Tables  3-4 
through  3-7  provide  counts  of  sampled  schools  by  the  primary  stratification  variables.  The 


DDESS  schools,  the  DoDDS  schools  (except  fourth  grade),  and  Guam  samples  are  not  included 
in  these  tables  as  all  schools  in  these  jurisdictions  were  sampled  with  certainty,  thereby  requiring 
no  stratification.  The  DoDDS  fourth-grade  sample  was  sorted  by  Department  of  Defense 
Education  Activity  (DoDEA)  area  (Europe,  Pacific,  etc.),  DoDEA  district  (Brussels,  Heidelberg, 
Italy,  etc,),  and  estimated  fourth-grade  enrollment  prior  to  sample  selection. 


3.4.2  Missing  Stratification  Variables 

As  stated  earlier,  the  sampling  frame  for  the  1996  State  Assessment  was  the  most  recent 
version  of  the  QED  file  available  combined  with  the  1993  PSS  list  of  nonpublic  schools.  The 
CCD  file  was  used  to  extract  information  on  urbanization  (“type  of  location”)  for  public  schools 
where  this  information  was  missing  on  the  QED  file.  Any  public  schools  with  missing  values 
remaining  in  urbanization  or  minority  enrollment  data  were  imputed. 

Schools  with  missing  values  in  urbanization  data  were  assigned  the  urbanization  of  other 
school  records  within  the  same  state,  county,  and  city  when  urbanization  did  not  vary  within  the 
given  city.  Any  schools  still  missing  urbanization  were  assigned  the  modal  value  of  urbanization 
within  their  city.  Any  remaining  missing  values  were  assigned  individually  based  on  city  using 
U.S.  Bureau  of  Census  publications. 

Schools  with  missing  values  in  minority  enrollment  data  were  assigned  the  average 
minority  enrollment  within  their  school  district.  Any  schools  still  missing  minority  enrollment 
data  were  assigned  values  individually  using  ZIP  code  and  U.S.  Bureau  of  Census  data.  The 
minority  data  were  extracted  only  for  those  schools  in  jurisdictions  in  which  minority 
stratification  was  performed. 

Metro  area  status  was  assigned  to  each  nonpublic  school  based  on  U.S.  Bureau  of  Census 
definitions  as  of  June  30,  1993,  based  on  Federal  Information  Processing  Standard  (FIPS)  county 
code,  and  was  found  for  all  schools  in  the  sampling  frame.  The  Catholic  school  flag  was  assigned 
to  each  nonpublic  school  based  on  the  QED  or  PSS  school  type  and  was  found  for  all  schools  in 
the  sampling  frame. 

Median  household  income  wai  assigned  to  every  school  in  the  sampling  frame  by 
merging  on  ZIP  code  with  a file  from  Donnelly  Marketing  Information  Services.  Any  schools 
still  missing  median  household  income  were  assigned  the  mean  value  of  median  household 
income  for  the  three-digit  ZIP  code  prefix  or  county  within  which  they  were  located. 


3.4,3  Urbanization  Classification 

Urbanization  classification  was  created  based  on  the  NCES  type  of  location  variable. 
The  type  of  location  variable  contains  at  most  seven  levels: 

1.  Large  Central  City : A central  city  of  a Metropolitan  Statistical  Area  (MSA) 
with  a population  greater  than  or  equal  to  400,000,  or  a population  density 
greater  than  or  equal  to  6,000  persons  per  square  mile; 
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2.  Mid-size  Central  City : A central  city  of  an  MSA  but  not  designated  as  a 
large  central  city; 

3.  Urban  Fringe  of  Large  City : A place  within  an  MSA  of  a large  central  city 
and  defined  as  urban  by  the  U.S.  Bureau  of  Census; 

4.  Urban  Fringe  of  Mid-Size  City : A place  within  an  MSA  of  a mid-size  central 
city  and  defined  as  urban  by  the  U.S.  Bureau  of  Census; 

5.  Large  Town:  A place  not  within  an  MSA,  but  with  a population  greater  than 
or  equal  to  25,000  and  defined  as  urban  by  the  U.S.  Bureau  of  Census; 

6.  Small  Town:  A place  not  within  an  MSA,  with  a population  less  than  25,000, 
but  greater  than  2,499  and  defined  as  urban  by  U.S.  Bureau  of  Census;  and 

7.  Rural:  A place  with  a population  of  less  than  2,500  and  defined  as  rural  by 
the  U.S.  Bureau  of  Census. 


Urbanization  classification  was  created  by  collapsing  type  of  location  categories  as 
necessary  and  according  to  specific  rules  until  each  urbanization  stratum  included  a minimum  of 
10  percent  of  eligible  students  in  the  participating  jurisdiction.  Tables  M and  1-2  in  Appendix  I 
provide  the  urbanization  classifications  used  within  each  jurisdiction  and  grade. 


3.4.4  Minority  Classification 

Minority  classification  was  created  within  urbanization  strata  and  was  based  on  a 
school’s  percentages  of  Black  and  Hispanic  students.  Three  different  minority  classification 
schemes  were  used  and  are  described  as  follows: 

• Case  I : Urbanization  strata  with  less  than  10  percent  Black  students  and  7 
percent  Hispanic  students  were  not  stratified  by  minority  enrollment 
(Level  0); 

• Case  2:  Urbanization  strata  with  greater  than  or  equal  to  10  percent  Black 
students  or  7 percent  Hispanic  students,  but  not  more  than  20  percent  of 
each,  were  stratified  by  ordering  percent  minority  enrollment  (Black  plus 
Hispanic)  within  the  urbanization  classes  and  dividing  the  schools  into  three 
groups  with  about  equal  numbers  of  students  per  minority  classification 
(Levels  1,  2,  and  3);  and 

• Case  3:  In  urbanization  strata  with  greater  than  20  percent  of  both  Black  and 
Hispanic  students,  minority  strata  were  formed  with  the  objective  of 
providing  equal  strata  with  emphasis  on  the  minority  group  (Black  or 
Hispanic)  with  the  higher  concentration.  The  stratification  was  performed  as 
follows.  The  minority  group  with  the  higher  percentage  gave  the  primary 
stratification  variable;  the  remaining  group  gave  the  secondary  stratification 
variable.  Within  urbanization  class,  the  schools  were  sorted  based  on  the 
primary  stratification  variable  and  divided  into  two  groups  of  schools 
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containing  approximately  equal  numbers  of  students  based  on  estimated 
grade  enrollment.  Within  each  of  these  two  groups,  the  schools  were  sorted 
by  the  secondary  stratification  variable  and  subdivided  into  two  subgroups  of 
schools  containing  approximately  equal  numbers  of  students.  As  a result, 
within  urbanization  strata  there  were  four  minority  classifications  (e.g.,  low 
Black/low  Hispanic,  low  Black/high  Hispanic,  high  Black/low  Hispanic,  and 
high  Black/high  Hispanic  (Levels  4,  5,  6,  and  7). 

The  minority  groups  and  classifications  were  formed  solely  for  the  purpose  of  creating 
efficient  stratification  design  at  this  stage  of  sampling.  These  classifications  are  not  directly  used 
in  analysis  and  reporting  of  the  data,  but  will  act  to  reduce  sampling  errors  for  achievement-level 
estimates.  Tables  1-1  and  1-2  in  Appendix  I provide  information  on  minority  stratification  for  the 
participating  jurisdictions. 


3.4.5  Median  Household  Income 


Prior  to  the  selection  of  the  school  samples,  the  public  schools  were  sorted  by  their  four 
stratification  variables  (small  or  large  district  status,  school  class  size,  urbanization  classification, 
and  minority  classification)  in  an  order  such  that  changes  occur  on  only  one  variable  at  a time 
(also  known  as  a serpentine  order.)  This  is  accomplished  by  alternating  between  ascending  and 
descending  sort  order  on  each  variable  successively  through  the  sort  hierarchy.  Within  this  sorted 
list,  the  schools  were  sorted,  in  serpentine  order,  by  the  median  household  income.  This  final 
stage  of  sorting  resulted  in  implicit  stratification  of  median  household  income.  The  data  on 
median  household  income  was  related  to  the  ZIP  code  area  in  which  the  school  is  located.  The 
data  were  derived  from  the  1990  Census  and  are  obtained  from  Donnelly  Marketing  Information 
Services. 


3.4.6  Metro  Area  Status 

All  schools  in  the  sampling  frame  were  assigned  a metro  area  status  based  on  their  FIPS 
county  code  and  Office  of  Management  and  Budget  (OMB)  Metropolitan  Area  Definitions  as  of 
June  30,  1993.  This  field  indicated  if  a school  was  located  within  a metropolitan  area  or  not. 
Tables  1-3  and  1-4  in  Appendix  I provide  information  on  metro  area  status  stratification  for  the 
participating  jurisdictions. 


3.4.7  School  Type 

All  nonpublic  schools  were  assigned  a school  type  (Catholic  or  other  nonpublic)  based 
on  their  QED  or  PSS  school-type  variable.  Tables  1-3  and  1-4  in  Appendix  I provide  information 
on  school-type  stratification  for  the  participating  jurisdictions. 


'i ) 
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3.5 


SCHOOL  SAMPLE  SELECTION 


3.5.1  Measure  of  Size  and  Sample  Selection 

Each  grade-eligible  school  was  assigned  an  estimated  grade  enrollment  by  dividing  its 
total  student  enrollment  by  its  number  of  grades.  Each  school  was  then  assigned  a measure  of 
size  based  on  the  following  function  of  estimated  grade  enrollment  (EGE).  Tables  3-4  and  3-5 
provide  the  estimated  grade  enrollment  and  measure  of  size  for  grades  4 and  8. 


Table  3-4 

Estimated  Grade  Enrollment  and  Measure  of  Size,  Grade  4 


Estimated  Grade  Enrollment 

Measure  of  Size 

EGE  < 10 

15 

10  < EGE  < 20 

1.5  ♦EGE 

20  * EGE  < 33 

30 

33  < EGE 

EGE 

Table  3-5 

Estimated  Grade  Enrollment  and  Measure  of  Size,  Grade  8 

Estimated  Grade  Enrollment 

Measure  of  Size 

EGE  < 10 

30 

10  a EGE  <20 

3 * EGE 

20  < EGE  < 65 

60 

65  < EGE 

EGE 

Schools  were  designated  as  being  in  “small”  or  “large”  districts  and  were  assigned  to  one 
of  two  size  classes  as  shown  in  Tables  3-4  and  3-5.  A large  district  was  defined  as  a district 
containing  20  percent  or  more  of  a jurisdiction's  student  population.  All  other  districts  were 
considered  small.  Schools  were  assigned  to  the  large  size  class  if  their  estimated  grade 
enrollment  was  greater  than  19.  Otherwise  schools  were  assigned  to  the  small  size  class. 

A sample  of  schools  was  then  selected  for  each  jurisdiction  with  probability  proportional 
to  each  school's  measure  of  size.  The  sampling  frame  of  schools  was  sorted  in  systematic  order 
prior  to  sample  selection,  as  follows: 

• Public  schools 

=>  Small  or  large  district  status. 

Size  class, 

=>  U than i city  stratum. 

Minority  stratum,  and 
:=>  Median  household  income. 


p » 
r»  i 
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• Nonpublic  schools 
=>  Size  class, 

=>  Metro  area  status, 

=>  Catholic/non  Catholic,  and 
=s>  Estimated  grade  enrollment. 

Sorting  the  sampling  frame  in  a specific  order  prior  to  systematic  sample  selection 
ensures  that  the  sampled  units  represent  a variety  of  population  subgroups. 

3.5.2  Control  of  Overlap  of  School  Samples  for  National  Educational  Studies 

The  issue  of  school  sample  overlap  has  been  relevant  in  all  rounds  of  NAEP  in  recent 
years.  To  avoid  undue  burden  on  individual  schools,  NAEP  developed  a policy  for  1996  of 
avoiding  overlap  between  national  and  state  samples.  This  was  to  be  achieved  without  unduly 
distorting  the  resulting  samples  by  introducing  bias  or  substantial  variance.  The  procedure  used 
was  an  extension  of  the  method  proposed  by  Keyfitz  (1951).  The  general  approach  is  given  in 
The  NAEP  1994  Technical  Report  (Allen,  Kline,  & Zelenak,  1996).  Counts  of  school  selection 
for  both  state  and  national  NAEP  are  found  in  Table  3-6. 


Table  3-6 

Number  of  Schools  Selected  for  Both  State  and  National  NAEP,  by  Grade  and  School  Type 


State  NAEP 
Grade  School  Type 

4 Main 

4 Trend 

National  NAEP  Grade 
8 Main  8 Trend  12  Main 

12  Trend 

4 

Public 

10 

29 

9 

4 

4 

0 

4 

Nonpublic 

0 

2 

17 

1 

11 

5 

8 

Public 

8 

4 

53 

101 

26 

4 

8 

Nonpublic 

23 

5 

5 

5 

22 

4 

3.5.3  Selection  of  Schools  in  Small  Jurisdictions 

All  schools  in  jurisdictions  with  small  numbers  of  public  schools  were  selected.  The 
jurisdictions  and  grades  are  shown  in  Table  3-7. 


Table  3-7 

Jurisdictions  Where  All  Public  Schools  were  Selected . by  Grade  and  School  Type 


Jurisdiction 

Public 

Grade  4 

Grade  8 

Nonpublic 

Grade  4 Grade  8 

Delaware 

* 

* 

District  of  Columbia 

* 

* 

* 

DoDEA7DDESS 

* 

* 

DoDEA/DoDDS 

* 

Guam 

* 

* 

* * 

Hawaii 

* 

Rhode  Island 

* 

3.5.4  New  School  Selection 

A sample  of  new  schools  was  drawn  to  properly  reflect  additions  to  the  target  population 
occurring  after  the  sampling  frame  building  information  was  created.  A district-level  file  was 
constructed  from  the  combined  QED  and  PSS  school-level  files.  The  district-level  file  was 
divided  into  a “small”  districts  file  that  was  not  used  in  the  selection  of  new  schools,  and  a 
“medium  and  large”  districts  files  that  was  used  for  this  purpose.  Small  districts  consisted  of 
those  districts  in  which  there  were  at  most  three  schools  on  the  aggregate  frame  and  no  more  than 
one  fourth-,  one  eighth-,  and  one  twelfth-grade  school.  The  remainder  of  districts  were  denoted 
as  “medium  and  large”  districts. 

A sample  of  medium  and  large  districts  was  drawn  in  each  jurisdiction.  All  districts  were 
selected  in  Delaware,  the  District  of  Columbia,  Hawaii,  and  Rhode  Island.  The  remaining 
jurisdictions  in  the  file  of  medium  and  large  districts  (eligible  for  sampling)  were  divided  into 
two  files  within  each  district.  Two  districts  were  selected  per  jurisdiction  with  equal  probability 
among  the  smaller  districts  with  combined  enrollment  of  less  than  or  equal  to  20  percent  of  the 
state  enrollment  in  the  medium  and  large  districts  file.  From  the  rest  of  the  file,  eight  districts 
were  selected  per  jurisdiction  with  probability  proportional  to  enrollment.  The  breakdown  given 
above  applied  to  all  jurisdictions  except  Alaska  and  Nevada,  w'here  four  and  seven  districts  were 
selected  with  equal  probability  and  six  and  three  districts  were  selected  with  probability 
proportional  to  enrollment,  respectively.  The  10  selected  districts  in  each  jurisdiction  were  then 
sent  a listing  of  all  their  schools  that  appeared  on  the  file,  and  w-ere  asked  to  provide  information 
about  the  new  schools  not  included  in  the  tile.  These  listings,  provided  by  selected  districts,  were 
used  as  sampling  frames  for  selection  of  new  schools. 


The  eligibility  of  a school  was  determined  based  on  the  grade  span.  A school  was  also 
classified  as  “new”  if  a change  of  grade  span  was  such  that  the  school  status  changed  from 
ineligible  to  eligible.  The  average  grade  enrollment  for  these  schools  was  set  to  the  average  grade 
enrollment  before  the  grade-span  change.  The  schools  found  eligible  for  sampling  due  to  the 
grade-span  change  were  added  to  the  new  school  selection  frame. 


The  probability  of  selecting  a school  was  minimum 


sampling  rate  * measure  of  si/.e 
/^(district) 


where  P(  district)  was  the  probability  of  selection  of  a district  and  the  sampling  rate  was  the  rate 
used  for  the  particular  jurisdiction  in  the  selection  of  the  original  sample  of  schools. 
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In  each  jurisdiction,  the  sampling  rate  used  for  the  main  sample  of  grade-eligible  schools 
was  used  to  select  the  new  schools.  Additionally,  all  new  eligible  schools  coming  from  small 
districts  (those  with  at  most  one  grade  4 and  one  grade  8 school  and  at  most  three  schools  on  the 
aggregate  frame)  that  had  a school  selected  in  the  regular  sample  for  the  fourth  grade  were 
included  in  the  sample  with  certainty.  In  the  1996  State  Assessment,  there  were  no  such  schools. 

Tables  3-8  and  3-9  shows  the  number  of  new  schools  coming  from  the  medium  and  large 
and  small  districts  for  the  fourth-  and  eighth-grade  samples. 


Table  3-8 

Distribution  of  New  Schools  Coming  from  “ Medium " or  "Large"  and  "Small"  Districts 

in  the  Fourth-Grade  Sample 


Jurisdiction 

Number  of  New  Schools 

“Medium”  or  “Large”  Districts  “Small”  Districts 

Alabama 

i 

0 

Alaska 

0 

0 

Arizona 

0 

0 

Arkansas 

0 

0 

California 

0 

0 

Colorado 

0 

0 

Connecticut 

0 

0 

Delaware 

1 

0 

DoDEA/DDESS 

0 

0 

DoDEA/DoDDS 

0 

{) 

District  of  Columbia 

1 

0 

Florida 

1 

0 

Georgia 

1 

0 

Guam 

0 

0 

Hawaii 

0 

0 

Indiana 

0 

0 

Iowa 

0 

0 

Kentucky 

1 

0 

Louisiana 

3 

0 

Maine 

i 

0 

Maryland 

0 

Massachusetts 

0 

Michigan 

0 

Minnesota 

0 

0 

Mississippi 

3 

0 

Missouri 

0 

0 

Montana 

0 

0 

Nebraska 

1 

0 

Nevada 

3 

0 

New  Jersev 

1 

0 
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Table  3-8  (continued) 

Distribution  of  New  Schools  Coming  from  “ Medium ” or  "Large”  and  "Small”  Districts  in  the 

Fourth-Grade  Sample 


Jurisdiction 

New  Mexico 

New  York 

North  Carolina 

North  Dakota 

Oregon 

Pennsylvania 

Rhode  Island 

South  Carolina 

Tennessee 

Texas 

Utah 

Vermont 

Virginia 

Washington 

West  Virginia 

Wisconsin 

Wyoming 

Total 


Number  of  New  Schools 

“Medium”  or  “Large”  Districts  “Small”  Districts 


0 

1 

3 


3 

1 

2 

1 

1 

3 

0 

0 

0 

0 

1 
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0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


Table  3-9 

Distribution  of  New  Schools  Coming  from  "Medium”  or  “Large  and  Small  Districts  in  the 

Eighth-Grade  Sample 


Jurisdiction 

Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

DoDEA/DDESS 

DoDEA/DoDDS 

District  of  Columbia 

Florida 

Georgia 

Guam 

Hawaii 

Indiana 

Iowa 


Number  of  New  Schools 

“Medium”  or  “Large”  Districts  “Small”  Districts 


0 

0 

1 

1 

0 

1 

0 

2 

0 

0 

2 

4 

2 

0 

1 

1 

0 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


Table  3-9  (continued) 

Distribution  of  New  Schools  Coming  from  “ Medium ” or  “Large”  and  “Small”  Districts 

in  the  Eighth-Grade  Sample 


Jurisdiction 

Number  of  New  Schools 

“Medium”  or  “Large”  Districts  “Small”  Districts 

Kentucky 

0 

0 

Louisiana 

3 

0 

Maine 

0 

0 

Maryland 

1 

0 

Massachusetts 

1 

0 

Michigan 

0 

0 

Minnesota 

0 

0 

Mississippi 

2 

0 

Missouri 

1 

0 

Montana 

1 

0 

Nebraska 

3 

0 

Nevada 

2 

0 

New  Hampshire 

0 

0 

New  Jersey 

0 

0 

New  Mexico 

0 

0 

New  York 

0 

0 

North  Carolina 

1 

0 

North  Dakota 

0 

0 

Oregon 

2 

0 

Rhode  Island 

0 

0 

South  Carolina 

1 

0 

Tennessee 

2 

0 

Texas 

1 

0 

Utah 

3 

0 

Vermont 

2 

0 

Virginia 

0 

0 

Washington 

1 

0 

West  Virginia 

0 

0 

Wisconsin 

3 

0 

Wyoming 

4 

0 

Total 

55 

0 

3.5.5  Assigning  Subject,  Sample  Type,  and  Monitor  Status 

Subject  assignment  rules  varied  by  grade.  All  fourth  grade  schools  were  assigned  to 
participate  in  mathematics  assessments  except  for  the  DDESS  and  DoDDS  samples  where  the 
rules  for  subject  assignment  at  eighth  grade  were  followed.  All  eighth-grade  schools  with  20  or 
more  students  were  assigned  to  participate  in  both  mathematics  and  science  assessments.  Schools 
with  less  than  20  students  were  assigned  one  subject  selected  at  random. 

The  1996  State  Assessment  used  two  different  sets  of  inclusion  rules  (see  Chapter  4)  for 
different  sets  of  schools  (SI  and  S2  subsamples).  A sample  type  variable  was  created  to  reflect 
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which  set  of  rules  to  use  within  a given  school.  The  sampled  schools  were  sorted  by  stratum 
(public  and  nonpublic)  and  subject  (both  mathematics  and  science,  mathematics  only,  and 
science  only)  and  then  randomly  assigned  sample  type  within  the  sorted  list.  The  sets  of  inclusion 
rules  are  described  in  Chapter  4. 

Jurisdictions  received  25  or  50  percent  monitoring  of  sessions  depending  on  previous 
participation  in  the  state  assessments.  All  jurisdictions  received  25  percent  monitoring  except 
Alaska,  Nevada,  Vermont,  and  Washington,  where  50  percent  monitoring  was  used.  The  sampled 
schools  were  sorted  by  stratum,  subject,  and  sample  type,  and  then  assigned  the  two  levels  of 
monitoring  at  random. 


3.5.6  School  Substitution  and  Retrofitting 

A substitute  school  was  assigned  to  each  sampled  school  (to  the  extent  possible)  prior  to 
the  field  period  through  an  automated  substitute  selection  mechanism  that  used  distance 
measures  as  the  matching  criterion.  Schools  were  also  required  to  be  of  the  same  type  (i.e., 
public,  nonpublic,  BIA,  and  DoDEA  schools  were  only  allowed  to  substitute  for  each  other),  and 
substitutes  for  nonpublic,  BIA,  and  DoDEA  schools  were  required  to  come  from  within  the  same 
district.  Public-school  substitutes  were  required  to  come  from  different  districts.  Two  passes 
were  made  at  the  substitution,  with  the  second  pass  raising  the  maximum  distance  measure 
allowed  and  removing  the  different  district  assignment  for  public  schools.  This  strategy  was 
motivated  from  the  fact  that  most  public-school  nonresponse  is  really  at  the  district  level. 

A distance  measure  was  used  in  each  pass  and  was  calculated  between  each  sampled 
school  and  each  potential  substitute.  The  distance  measure  was  equal  to  the  sum  of  four  squared 
standardized  differences.  The  differences  were  calculated  between  the  sampled  and  potential 
substitute  school’s  estimated  grade  enrollment,  median  household  income,  percent  Black 
enrollment  and  percent  Hispanic  enrollment.  Each  difference  was  squared  and  standardized  to 
the  population  standard  deviation  of  the  component  variable  (e.g.,  estimated  grade  enrollment) 
across  all  grade-eligible  schools  and  jurisdictions.  The  potential  substitutes  were  then  assigned  to 
sampled  schools  by  order  of  increasing  distance  measure.  An  acceptance  limit  was  put  on  the 
distance  measure  of  60  for  the  first  pass.  A given  potential  substitute  was  assigned  to  one  and 
only  one  sampled  school.  Some  sampled  schools  did  not  receive  assigned  substitutes  (at  least  in 
the  first  pass)  because  the  number  of  potential  substitutes  was  less  than  the  number  of  sampled 
schools  or  the  distance  measure  for  all  remaining  potential  substitutes  from  different  districts  was 
greater  than  .60. 

In  the  second  pass,  th  different  district  constraint  for  public  schools  was  lifted  and  the 
maximum  distance  allowed  was  raised  to  .75.  This  generally  brought  in  a small  number  of 
additional  assigned  substitutes.  Although  the  selected  cut-off  points  of  .60  and  .75  on  the 
distance  measure  were  somewhat  arbitrary,  they  had  been  decided  upon  for  the  1994  Trial  State 
Assessment  by  a group  of  statisticians  reviewing  a large  number  of  listings  beforehand  and 
finding  a consensus  on  the  distance  measures  at  which  substitutes  began  to  appear  unacceptable. 


Jurisdictions  that  did  not  receive  substitutes  for  all  selected  schools  were  allowed  to 
retrofit  unused  substitutes  after  part  of  the  field  period  elapsed.  Substitutes  that  were  assigned  to 
cooperating  or  ineligible  original  selections  were  free  to  be  assigned  to  other  original  selections 
that  did  not  receive  substitutes.  These  free  substitutes  were  put  back  into  the  substitute  selection 
mechanism  described  above  and  allowed  to  pair  up  with  other  original  selections. 

Cooperating  original  selections  were  also  allowed  to  serve  as  “double  session” 
substitutes  for  other  pending  or  refusing  schools  and  were  put  through  the  substitute  selection 
mechanism  after  retrofitting  unused  regular  substitutes.  Double  session  substitutes  are 
particularly  helpful  to  small  jurisdictions  where  all  or  most  schools  are  taken  as  original 
selections,  thereby  leaving  no  or  few  schools  available  as  substitutes. 

Tables  1-5  and  1-6  in  Appendix  I include  information  about  the  number  of  substitutes 
provided  in  each  jurisdiction.  Of  the  47  participating  jurisdictions,  42  were  provided  with  at  least 
one  substitute  at  grade  4,  and  41  were  provided  with  at  least  one  substitute  at  grade  8.  Among 
jurisdictions  receiving  no  substitutes,  the  majority  had  100  percent  participation  from  the  original 
sample.  The  number  of  substitutes  provided  to  a jurisdiction  ranged  from  0 to  24  in  the  fourth- 
grade  sample.  A total  of  243  substitutes  were  selected. 

Tables  1-7  through  I- 10  in  Appendix  I show  the  number  of  schools  in  the  fourth-  and 
eighth-grade  mathematics  samples,  together  with  school  response  rates  observed  within 
participating  jurisdictions.  The  tables  also  show  the  number  of  substitutes  in  each  jurisdiction 
that  were  associated  with  a nonparticipating  original  school  selection,  and  the  number  of  those 
that  participated. 


3.6  STUDENT  SAMPLE  SELECTION 
3.6.1  Student  Sampling  and  Participation 

Schools  initially  sent  a complete  list  of  students  to  a central  location  in  November  1995. 
They  were  not  asked  to  list  students  in  any  particular  order,  but  were  asked  to  implement  checks 
to  ensure  that  all  grade-eligible  students  were  listed.  Based  on  the  total  number  of  students  on 
this  list,  the  “Student  Listing  Form,”  sample  line  numbers  were  generated  for  student  sample 
selection.  To  generate  tt.  sse  line  numbers,  the  sampler  entered  the  number  of  students  on  the 
form  and  the  number  of  sessions  into  a calculator  or  personal  computer  that  had  been 
programmed  with  the  sampling  algorithm.  The  program  generated  a random  start  that  was  used  to 
systematically  select  the  student  line  numbers  (30  per  session).  To  compensate  for  new  enrollees 
not  on  the  Student  Listing  Form,  extra  line  numbers  were  generated  for  a supplemental  sample  of 
new  students. 

After  the  student  sample  was  selected,  the  administrator  at  each  school  identified 
students  who  were  incapable  of  taking  the  assessment  either  because  they  were  identified  as 
students  with  disabilities  (SD)  or  because  they  were  classified  as  being  of  limited  English 
proficiency  (LEP).  Two  different  sets  of  inclusion  rules  were  used:  a set  used  in  previous 
assessments  and  a new  set  that  was  meant  to  clarify  the  inclusion  rules  used  in  NAEP  and  to 
provide  wider  inclusion  of  SD  and  LEP  students.  More  details  on  the  procedures  for  student 
exclusion  are  presented  in  the  report  on  field  procedures  for  the  1996  State  Assessment  program 
(Westat,  Inc.,  1996). 


When  the  assessment  was  conducted  in  a given  school,  a count  was  made  of  the  number 
of  nonexcluded  students  who  did  not  attend  the  session.  If  this  number  exceeded  three  students, 
the  school  was  instructed  to  conduct  a makeup  session,  to  which  all  students  who  were  absent 
from  the  initial  session  were  invited. 

Tables  1-1 1 through  I- 14  in  Appendix  I provide  the  distribution  of  the  student  samples 
and  response  rates  by  grade,  school  type,  and  jurisdiction. 


3.6.2  The  Reduced  Sample  Option 

All  jurisdictions  were  given  the  option  to  reduce  the  expected  student  sample  size  in 
order  to  reduce  testing  burden  and  the  number  of  multiple-testing  sessions  for  participating 
schools.  If  jurisdictions  chose  to  exercise  this  option,  the  estimates  obtained  from  the  assessment 
were  more  variable  than  they  otherwise  would  have  been.  In  general,  jurisdictions  could  reduce 
student  sample  sizes  by  adjusting  the  number  of  sessions  with  participating  schools  subject  to  the 
following  constraints: 

• The  minimum  number  of  sessions  per  school  had  to  be  equal  to  1 ; 

• The  maximum  number  of  sessions  per  school  had  to  be  equal  to  2 at  fourth 
grade  and  3 at  eighth  grade; 

• The  expected  student  size  from  the  reduced  sample  was  greater  than  or  equal 
to  half  of  the  original  student  sample  size. 

Table  3-10  shows  the  jurisdictions  that  exercised  the  reduced  sample  option  at  each 

grade. 


Table  3-10 

Jurisdictions  Exercising  the  Reduced  Sample  Option , By  Grade 


Jurisdiction 

Grade  4 

Grade  8 

Alaska 

* 

Delaware 

* 

* 

Guam 

* 

Hawaii 

* 

Rhode  Island 

* 

Chapter  4 


STATE  AND  SCHOOL  COOPERATION  AND  FIELD 
ADMINISTRATION1 

Lucy  M.  Gray 
Westat9  Inc . 


4.1  OVERVIEW 

By  volunteering  to  participate  in  the  State  Assessment  and  in  the  field  test  that  preceded 
it,  each  jurisdiction  assumed  responsibility  for  securing  the  cooperation  of  the  schools  sampled 
by  NAEP.  The  participating  jurisdictions  were  responsible  for  the  actual  administration  of  the 
1996  State  Assessment  at  the  school  level.  The  1995  field  test,  however,  operated  within  the 
framework  of  the  national  (rather  than  state)  model.  Therefore,  for  the  field  test,  NAEP  field  staff 
were  responsible  for  securing  cooperation  for,  scheduling,  and  conducting  the  assessments.  This 
chapter  describes  state  and  school  cooperation  and  field  administration  procedures  for  both  the 
1995  field  test  and  the  1996  assessment  program.  Section  4.2  presents  information  on  the  field 
test,  while  Section  4.3  focuses  on  the  1996  State  Assessment. 


4.2  THE  FIELD  TEST 
4.2.1  Conduct  of  the  Field  Test 

In  preparation  for  the  1996  state  and  national  assessment  programs,  a field  test  of  the 
forms,  procedures,  and  booklet  items  was  held  in  late  January  through  early  March  1995.  In  this 
field  test,  assessments  were  piloted  in:  mathematics,  science,  and  the  arts  (dance,  music,  theater, 
and  visual  arts).  In  an  effort  to  increase  the  participation  of  limited  English  proficient  (LEP) 
students  and  students  with  disabilities  (SD),  the  mathematics  field  test  included  bilingual  and 
Spanish-language  versions  of  three  test  booklets,  newly  developed  Braille  and  large-print 
booklets,  and  the  provision  of  additional  testing  accommodations  for  students  with  disabilities 
and  students  with  limited  English  proficiency.  Results  for  the  field  testing  of  the  Spanish- 
language  mathematics  assessment,  Braille  and  large-print  booklets,  and  special  testing 
accommodations  are  contained  in  a separate  report  prepared  by  Educational  Testing  Service 
(ETS)  (Anderson,  Jenkins,  Sc  Miller,  1996). 

A number  of  new  complexities  were  planned  for  the  1996  assessment,  such  as  increased 
use  of  manipulatives  in  mathematics,  theme  blocks  in  mathematics,  hands-on  tasks  in  science, 
and  performance  items  in  dance,  music,  theater,  and  visual  arts.  The  complexities  of  mathematics 
and  science  substantially  increased  the  scope  of  the  1996  assessment,  as  originally  defined,  and 
were  rehearsed  as  part  of  the  field  test. 


1 Lucy  Gray  was  responsible  for  \ annus  aspects  ol  the  hold  activities  lot  the  NAM’  national  and  State  Assessments  in 
mathematics. 
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In  September  1994,  letters  were  sent  from  the  U.S.  Department  of  Education  to  all  Chief 
State  School  Officers  inviting  them  to  participate  in  the  1995  field  test  of  materials  and 
procedures.  In  an  effort  to  secure  the  participation  of  more  schools  and  to  lessen  the  burden  of 
participation  on  jurisdictions,  ETS  and  Westat  offered  to  perform  all  of  the  work  involved, 
including  sampling,  communicating  with  school  staff,  and  administering  the  assessment. 

The  school  sample  for  the  field  test  included  both  public  and  nonpublic  schools  and  was 
designed  to  involve  as  many  states  as  possible,  thus  limiting  the  burden  on  each  state.  However, 
states  with  small  numbers  of  schools  in  which  all  schools  were  already  involved  in  the  1994 
National  Assessment  program  were  excluded  from  the  field  test  sample.  As  a result,  the  original 
field  test  sample  consisted  of  1,129  public  and  nonpublic  schools  spread  roughly  in  proportion  to 
the  population  across  38  states.  Because  the  states’  responsibilities  were  very  limited  in  the  field 
test,  they  were  asked  only  to  notify  districts  of  their  inclusion,  and  to  indicate  their  support  for 
participation  in  the  field  test.  Schools  selected  for  the  1995  field  test  were  designated  to  have 
either  arts  sessions  or  mathematics  and  science  sessions,  but  not  both. 

Because  the  focus  of  the  field  test  was  to  have  as  many  schools  participate  as  possible, 
flexibility  was  allowed  in  substituting  for  the  original  selections.  Three  forms  of  substitution 
were  available  to  replace  sampled  schools  that  did  not  participate  in  the  field  test.  The  first  type 
were  schools  identified  by  Westat  and  located  within  the  same  district  as  the  originally  sampled 
schools.  These  substitute  schools  were  demographically  comparable  to  those  in  the  original 
sample.  A second  school  substitution  option  allowed  district  superintendents  to  choose  their  own 
alternate  school.  In  the  event  that  a district  refused  to  participate,  the  third  option  was  an  “out  of 
district”  substitute,  identified  by  Westat.  The  type  and  number  of  sessions  scheduled  for  an 
originally  selected  school  were  carried  over  to  the  substitute  school. 

During  the  period  from  October  to  December  1994,  all  districts  and  schools  in  the  field 
test  sample  were  contacted,  cooperation  secured,  and  assessment  schedules  set.  To  accomplish 
these  initial  tasks,  21  of  the  most  experienced  NAEP  supervisors  were  trained  during  a three-day 
session  (in  early  October  1994)  conducted  by  Westat  project  staff.  Following  training,  each  of 
the  supervisors  was  responsible  for  scheduling  activities  in  several  states.  In  December  1994,  the 
NAEP  field  staff  was  expanded  to  72  supervisors.  All  supervisors,  including  those  in  the  original 
group,  attended  the  second  training  session.  After  opening  plenary  sessions,  the  trainees  were 
divided  into  two  groups:  arts  and  mathematics/science.  Because  of  the  complicated  nature  of  the 
arts  field  test,  it  was  decided  to  have  supervisors  specialize  in  the  administration  of  either  arts  or 
mathematics/science  sessions.  Training  focused  on  a review  of  the  scheduling  activities  during 
the  fall  (e.g.,  results  of  initial  contacts  with  districts  and  schools);  sampling  procedures; 
preparation  and  distribution  of  school,  teacher,  and  student  questionnaires;  administration  of  the 
performance-based  arts  tasks;  classroom  management  techniques;  exercise  administrator 
training;  and  completion  of  administrative  forms  and  procedures. 

The  period  from  January  2-20,  1995,  was  set  aside  for  supervisors  to  call  and  visit  the 
schools  in  their  assignments,  draw  student  samples,  prepare  Administration  Schedules,  and 
prepare  and  distribute  teacher,  school,  and  SD/LEP  student  questionnaires.  Assessments  were 
conducted  during  the  period  from  January  23  through  March  10,  1995.  Mathematics  and  science 
sessions  were  scheduled  to  be  completed  by  February  24,  and  arts  sessions  continued  through  the 
end  of  the  data  collection  period.  Throughout  the  field  testing  period,  supervisors  reported 
directly  to  Westat’s  field  director  through  six  field  managers. 


4.2.2  Results  of  the  Field  Test 


A total  of  963  originally  selected  schools  and  alternates  actually  participated  in  the  field 
test.  The  final  assessed  sample  of  schools  included  434  schools  at  grade  4,  395  schools  at  grade 
8,  and  134  schools  at  grade  12. 

A total  of  46,514  students  participated  in  the  field  test.  Of  this  number,  17,212  students 
participated  in  the  1995  arts  field  test;  these  students  will  be  discussed  in  a later  report  on  the  arts 
assessment.  Student  participation  in  mathematics  and  science  included  1 1,014  students  at  grade 
4,  1 1,641  students  at  grade  8,  and  6,647  students  at  grade  12. 


4.3  THE  1996  STATE  ASSESSMENT 

Forty-four  states,  the  District  of  Columbia,  and  Guam  volunteered  for  the  1996  State 
Assessment,  as  did  the  Department  of  Defense  Domestic  Dependent  Elementary  and  Secondary 
Schools  (DDESS)  and  the  Department  of  Defense  Dependents  Schools  (DoDDS).  Table  4-1 
identifies  the  jurisdictions  participating  in  the  Stale  Assessment. 

Table  4-1 

Jurisdictions  Participating  in  the  1996  State  Assessment  Program  in  Mathematics 


Jurisdictions 


Alabama 

Georgia 

Mississippi2 

Pennsylvania^ 

Alaska1 

Guam 

Missouri2 

Rhode  Island 

Arizona 

Hawaii 

Montana3 

South  Carolina2 

Arkansas 

Indiana 

Nebraska 

Tennessee2 

California 

Iowa 

Nevada1 

Texas 

Colorado 

Kentucky 

New  Hampshire5 

Utah2 

Connecticut 

Louisiana 

New  Jersey 

Vermont1 

Delaware 

Maine2 

New  Mexico 

Virginia 

DoDEA/DDESSl/6 

Maryland 

New  York 

Washington1 

DoDEA/DoDDS 1 lb 

Massachusetts2 

North  Carolina 

West  Virginia 

District  of  Columbia 

Michigan 

North  Dakota 

Wisconsin 

Florida 

Minnesota 

Oregon3 

Wyoming 

'Participated  in  the  1996  mathematics  assessment  program  only. 

Participated  in  the  1992  and  1996  mathematics  assessment  programs  but  not  in  the  1990  program. 

Participated  in  the  1990  and  1996  mathematics  assessment  programs  but  not  in  the  1992  program. 

4Gradc  4 only. 

Pjrade  8 only. 

flDoDEA  is  the  Department  of  Defense  Education  Activity  schools,  DDESS  is  the  Department  of  Defense  Domestic 
Dependent  Elementary  and  Secondary  Schools,  and  DoDDS  is  the  Department  of  Defense  Dependents  Schools. 
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4.3.1  Overview  of  Responsibilities 


Data  collection  for  the  1996  State  Assessment  involved  a collaborative  effort  between 
the  participating  jurisdictions  and  the  NAEP  contractors,  especially  Westat,  the  field 
administration  contractor.  Westat' s responsibilities  included: 

• selecting  the  sample  of  schools  and  students  for  each  participating 
jurisdiction; 

• developing  the  administration  procedures  and  manuals; 

• training  state  personnel  to  conduct  the  assessments;  and 

• conducting  an  extensive  quality  assurance  program. 

Each  jurisdiction  volunteering  to  participate  in  the  1996  program  was  asked  to  appoint  a 
state  coordinator.  In  general,  the  coordinator  was  the  liaison  between  NAEP/Westat  staff  and  the 
participating  schools.  In  particular,  the  state  coordinator  was  asked  to: 

• gain  the  cooperation  of  the  selected  schools; 

• assist  in  the  development  of  the  assessment  schedule; 

• receive  the  lists  of  all  grade-eligible  students  from  the  schools; 

• coordinate  the  flow  of  information  between  the  schools  and  NAEP; 

• provide  space  for  the  Westat  state  supervisor  to  use  when  selecting  the 
sample  of  students; 

• notify  assessment  administrators  about  training  and  send  them  their  manuals; 
and 

• send  the  lists  of  sampled  students  to  the  schools. 

At  the  school  level,  an  assessment  administrator  was  responsible  for  preparing  for  and 
conducting  the  assessment  session(s)  in  one  or  more  schools.  These  individuals  were  usually 
school  or  district  staff  and  were  trained  by  Westat  staff.  The  assessment  administrator’s 
responsibilities  included: 

• receiving  the  list  of  sampled  students  from  the  state  coordinator; 

• identifying  sampled  students  who  should  be  excluded; 

• distributing  assessment  questionnaires  to  appropriate  school  staff  and 
collecting  them  upon  their  completion; 

• notifying  sampled  students  and  their  teachers; 

• administering  the  assessment  sessions(s); 

• completing  assessment  forms;  and 

• preparing  the  assessment  materials  for  shipment. 

Decisions  on  exclusion  were  made  in  consultation  with  school  staff  and  were  guided  by  the 
SD/LEP  questionnaires  completed  by  the  school  staff. 
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Westat  also  hired  and  trained  a state  supervisor  for  each  jurisdiction.  The  1996  State 
Assessment  involved  about  the  same  number  of  state  supervisors  (Westat  staff)  as  both  the  1992 
and  1994  assessments,  since  approximately  the  same  number  of  jurisdictions  were  involved  each 
year.  In  addition,  three  troubleshooters  were  trained  in  case  any  state  supervisor  was  unable  to 
complete  their  assignment.  The  primary  tasks  of  the  state  supervisor  were  to: 

• select  the  samples  of  students  to  be  assessed; 

• recruit  and  hire  the  quality  control  monitors  throughout  their  jurisdiction; 

• conduct  in-pcrson  assessment  administration  training  sessions;  and 

• coordinate  the  monitoring  of  the  assessment  sessions  and  makeup  sessions. 

Westat  hired  and  trained  six  field  managers  for  the  State  Assessment.  Each  field  manager 
was  responsible  for  working  with  the  state  coordinators  of  seven  to  eight  jurisdictions  and  for 
overseeing  assessment  activities.  The  primary  tasks  of  the  field  managers  were  to: 

• obtain  information  about  cooperation  and  scheduling; 

• make  sure  the  arrangements  for  the  assessments  were  set  and  assessment 
administrators  identified;  and 

• schedule  the  assessment  administrators  training  sessions. 

In  addition,  Westat  hired  between  four  and  six  quality  control  monitors  in  each 
jurisdiction  to  monitor  assessment  sessions. 


4*3*2  Schedule  of  Data  Collection  Activities 

Mid-September  1995  Westat  sent  lists  of  sampled  schools  for  the  national  and  state 

assessments  and  informational  materials  to  the  state  coordinators. 

October  1995  Westat  field  managers  visited  individual  jurisdictions  to  explain 

the  computerized  state  coordinator  system,  which  was  used  to 
keep  track  of  assessment- related  activities. 

September  - December  1995  State  coordinators  obtained  cooperation  from  districts  and 

schools.  State  coordinators  reported  participation  status  to  Westat 
field  managers  via  computer  files  or  printed  lists.  State 
coordinators  sent  student  listing  forms  and  supplemental  student 
listing  forms  to  participating  schools. 

November  9 - 12,  1995  State  supervisor  training. 

November  17,  1995  Suggested  cutoff  for  decisions  on  school  participation  and 

submission  of  lists  of  grade-eligible  students  to  state  coordinators 
for  sampling  purposes. 
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December  4-  15,  1995 


December  15,  1995 


December  15,  1995  - 
January  15,  1996 


January  4-6,  1996 
January  9 - 26,  1996 


NAEP  supervisor  visited  state  coordinators’  offices  to  select 
student  samples  and  prepare  Administration  Schedules  listing  the 
students  selected  for  each  sample. 

Westat  delivered  training  session  schedule  and  copies  of 
assessment  administrator  manuals  to  state  coordinators  for 
distribution. 

State  coordinators  notified  assessment  administrators  of  the  date, 
time,  and  location  of  training  and  sent  each  a copy  of  the  manual 
for  assessment  administrators. 

Training  session  for  quality  control  monitors. 

Supervisors  conducted  assessment  administrator  training  sessions 
throughout  respective  jurisdictions. 


January  29  - March  1,  1996  Assessments  conducted  and  monitored. 


March  4 - 8,  1996  Makeup  week  for  rescheduled  assessments  or  completed 

assessments  requiring  makeup. 


4.3.3  Preparations  for  the  State  Assessment 

The  focal  point  of  the  schedule  for  the  State  Assessment  was  the  period  between  January 
29  and  March  4,  1996,  when  the  assessments  were  conducted  in  the  schools.  However,  as  with 
any  undertaking  of  this  magnitude,  the  project  required  many  months  of  planning  and 
preparation. 

Westat  selected  the  samples  of  schools  according  to  the  procedures  described  in 
Chapter  3.  In  mid-September  1995,  lists  of  the  selected  schools  and  other  materials  describing 
the  State  Assessment  program  were  sent  to  state  coordinators.  Most  state  coordinators  preferred 
that  NAEP  provide  a suggested  assessment  date  for  each  school.  School  listings  were  updated 
with  this  information  and  were  sent  to  the  state  coordinators,  along  with  other  descriptive 
materials  and  forms,  by  December. 

State  coordinators  were  also  given  the  option  of  receiving  the  school  information  in  the 
form  of  a computer  database  with  accompanying  management  information  software.  This  system 
enabled  state  coordinators  to  keep  track  of  the  cooperating  schools,  the  assessment  schedule,  the 
training  schedule,  and  the  assessment  administrators.  Coordinators  could  choose  to  receive  a 
laptop  computer  and  printer  or  to  have  the  system  installed  on  their  own  computer.  Westat  field 
managers  traveled  to  the  state  offices  to  explain  the  computer  system  to  the  state  coordinators 
and  their  staff.  Only  one  jurisdiction  chose  not  to  use  the  computerized  system.  In  this  case,  the 
state  coordinator  kept  track  of  information  on  logs  and  lists  provided  by  Westat.  This  printed 
information  was  mailed  to  the  field  manager  and  dictated  during  a regularly  scheduled  telephone 
conversation.  The  field  manager  then  entered  the  data  into  the  computer  database,  the  data  were 
transmitted  to  Westat,  and  reports  were  produced. 


60 


7 


Six  of  the  most  experienced  NAEP  supervisors  served  as  field  managers,  the  primary 
link  between  NAEP  and  the  state  coordinators.  During  late  summer  and  early  fall  1995,  the  field 
managers  received  copies  of  all  materials  sent  to  state  coordinators,  developed  a preliminary 
Assessment  Schedule  for  all  schools  in  their  jurisdictions,  and  became  thoroughly  familiar  with 
the  computer  system.  As  liaisons  with  the  state  coordinators,  they  visited  each  jurisdiction  to 
train  staff  in  the  use  of  the  computer  system.  Later  in  the  project  schedule,  they  attended  training 
sessions  for  the  supervisors  and  quality  control  monitors  and  also  presented  some  of  the  training 
material  at  each  of  these  sessions. 

The  field  managers  used  the  same  computer  system  as  the  state  coordinators  to  keep 
track  of  the  schools  and  the  schedule.  The  state  coordinators  sent  updates  via  computer  disks, 
telephone,  or  print  to  their  field  manager,  who  then  entered  the  information  into  the  system. 
Weekly  transmissions  were  made  from  the  field  manager  to  Westat. 

By  November,  Westat  had  hired  one  state  supervisor  for  each  participating  jurisdiction. 
The  state  supervisors  attended  a training  session  held  November  9-12,  1995.  This  training 
session  focused  on  the  state  supervisors’  immediate  tasks — selecting  the  student  samples  and 
hiring  quality  control  monitors.  Supervisors  were  given  the  training  script  and  materials  for  the 
assessment  administrators’  training  sessions  they  would  conduct  in  January  so  they  could 
become  familiar  with  these  materials. 

The  state  supervisors’  first  task  after  training  was  to  complete  the  selection  cf  the  sample 
of  students  who  were  to  be  assessed  in  each  school.  All  participating  schools  were  asked  to  send 
a list  of  their  grade-eligible  students  to  the  state  coordinator  by  November  17.  Sample  selection 
activities  were  conducted  in  the  state  coordinator’s  office  unless  the  state  coordinator  preferred 
that  the  lists  be  taken  to  another  location. 

Using  a preprogrammed  calculator,  the  supervisors  generally  selected  a sample  of  30 
students  per  session  type  per  school  with  three  exceptions:  in  schools  with  fewer  than  30  students 
in  the  grade  to  be  assessed,  all  of  the  students  were  selected;  in  schools  in  which  more  than  one 
session  was  scheduled,  60  students  (or  some  multiple  of  30  students)  were  selected;  and  in 
schools  with  no  more  than  33  students  in  the  grade,  all  students  were  selected  for  the  assessment. 

After  the  sample  was  selected,  the  supervisor  completed  an  Administration  Schedule  for 
each  session,  listing  the  students  to  be  assessed.  The  Administration  Schedules  for  each  school 
were  put  into  an  envelope  and  given  to  the  state  coordinator  to  send  to  the  school  two  weeks 
before  the  scheduled  assessment  date.  Included  in  the  envelope  were  instructions  for  sampling 
students  who  had  enrolled  at  the  schools  since  the  creation  of  the  original  list. 

During  the  months  of  November  and  December  1995,  the  state  supervisors  also  recruited 
and  hired  quality  control  monitors  to  work  in  their  jurisdictions.  It  was  the  quality  control 
monitor’s  job  to  observe  the  sessions  designated  to  be  monitored,  to  complete  an  observation 
form  on  each  session,  and  to  intervene  when  the  correct  procedures  were  not  followed.  Because 
earlier  results  indicated  little  difference  in  performance  between  monitored  and  unmonitored 
schools,  and  in  an  effort  to  reduce  costs,  the  percentage  of  public  schools  to  be  monitored  was 
maintained  at  25  percent  (i.e.,  the  reduced  monitoring  rate  initiated  in  1994).  The  monitoring  rate 
for  nonpublic  schools  was  also  reduced  to  25  percent  (from  50%  in  1994,  which  was  the  first 
year  that  nonpublic  schools  were  assessed  by  NAEP).  As  has  been  customary  in  the  past, 
monitoring  was  conducted  at  50  percent  lor  jurisdictions  that  were  new  to  the  Slate  Assessment 
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in  1996.  The  schools  to  be  monitored  were  known  only  to  contractor  staff;  it  was  not  indicated 
on  any  of  the  listings  provided  to  state  staff. 

Approximately  400  quality  control  monitors  were  trained  in  a session  held  in  early 
January  1996.  The  first  day  of  the  training  session  was  devoted  to  a presentation  of  the 
assessment  administrators'  training  program  by  the  state  supervisors,  which  not  only  gave  the 
monitors  an  understanding  of  what  assessment  administrators  were  expected  to  do,  but  gave  state 
supervisors  an  opportunity  to  practice  presenting  the  training  program.  The  remaining  days  of  the 
training  session  were  spent  reviewing  the  quality  control  monitor  observation  form  and  the  role 
and  responsibilities  of  the  quality  control  monitors. 

Almost  immediately  following  the  quality  control  monitor  training,  supervisors  began 
conducting  training  for  assessment  administrators.  Each  quality  control  monitor  attended  at  least 
two  training  sessions,  to  assist  the  state  supervisor  and  to  become  thoroughly  familiar  with  the 
assessment  administrator's  responsibilities.  Most  jurisdictions  had  approximately  14  training 
sessions  in  which  approximately  217  assessment  administrators  were  trained.  Almost  10,400 
assessment  administrators  were  trained  by  the  time  assessments  began  on  January  29,  1996. 

To  ensure  uniformity  in  the  training  sessions,  Westat  developed  a highly  structured 
program  involving  a script  for  trainers,  a videotape,  and  an  example  to  be  completed  by  the 
trainees.  The  training  package,  developed  for  previous  state  assessments,  was  revised  to  reflect 
the  subjects  and  grades  assessed  in  1996.  The  supervisors  were  instructed  to  read  the  script 
verbatim  as  they  proceeded  through  the  training,  ensuring  that  each  trainee  received  the  same 
information.  The  script  was  supplemented  by  the  use  of  overhead  transparencies,  displaying  the 
various  forms  that  were  to  be  used  and  enabling  the  trainer  to  demonstrate  how  they  were  to  be 
filled  out. 

The  videotape  was  also  revised  from  previous  versions  to  include  information  about 
assessing  both  fourth-  and  eighth-grade  students.  The  1996  version  of  the  video  ran  just  over  one 
hour. 


All  of  the  information  presented  in  the  training  session  was  included  in  Westat’s  Manual 
for  Assessment  Administrators . Copies  of  the  manual  were  sent  by  Westat  to  the  state 
coordinators  by  December  15,  1995,  so  that  they  could  be  distributed  to  the  assessment 
administrators  before  the  training  sessions.  The  method  of  distribution  and  the  amount  of  time 
that  the  assessment  administrators  had  to  study  the  manual  probably  varied  from  jurisdiction  to 
jurisdiction.  The  majority  of  the  assessment  administrators  appeared  to  have  become  at  least 
somewhat  familiar  with  the  manual  prior  to  their  training.  The  training  stressed  that  answers  to 
all  questions  about  procedures  or  forms  could  be  found  in  the  manual.  In  addition,  assessment 
administrators  were  provided  w ith  a toll-free  number  that  could  be  used  to  contact  Westat  if  they 
had  any  procedural  questions  or  were  in  need  of  additional  materials.  Dm  mg  the  assessment 
period,  this  telephone  number  was  used  extensively. 

The  entire  training  session  generally  ran  for  about  one-half  day  until  2 p.m.  including 


lunch. 


4.3.4  Monitoring  of  Assessment  Activities 

Two  weeks  prior  to  the  scheduled  assessment  date,  the  assessment  administrator  received 
the  Administration  Schedule  and  assessment  questionnaires  and  materials.  Five  days  before  the 
assessment,  the  quality  control  monitor  made  a cali  to  the  administrator  and  recorded  the  results 
of  the  call  on  the  Quality  Control  Form  for  Monitored  Schools  because  the  assessment 
administrators  were  not  supposed  to  know  in  advance  which  sessions  were  designated  to  be 
monitored.  The  pre-assessment  call  was  conducted  in  exactly  the  same  way  regardless  of  whether 
the  school  was  to  be  monitored  or  not.  For  example,  directions  to  the  school  were  obtained  even 
if  the  school  was  in  the  unmonitored  sample.  Most  of  the  questions  asked  in  the  pre-assessment 
call  were  designed  to  gauge  whether  the  assessment  administrator  had  received  all  materials 
needed  and  had  completed  the  preparations  for  the  assessment. 

If  the  sessions  in  a school  were  designated  to  be  monitored,  the  quality  control  monitor 
was  to  arrive  at  the  school  one  hour  before  the  scheduled  beginning  of  the  assessment  to  observe 
preparations  for  the  assessment.  To  ensure  the  confidentiality  of  the  assessment  items,  the 
booklets  were  packaged  in  shrink-wrapped  bundles  and  were  not  to  be  opened  until  the  quality 
control  monitor  arrived  or  45  minutes  before  the  session  began,  whichever  occurred  first. 

In  addition  to  observing  the  opening  of  the  bundles,  the  quality  control  monitor  used  the 
Quality  Control  Form  to  check  that  the  following  had  been  done  correctly:  sampling  newly 
enrolled  students,  reading  the  script,  distributing  and  collecting  assessment  materials,  timing  the 
booklet  sections,  answering  questions  from  students,  and  preparing  assessment  materials  for 
shipment.  After  the  assessment  was  over,  the  quality  control  monitor  obtained  the  assessment 
administrator’s  opinions  of  how  the  session  went  and  how  well  the  materials  and  forms  worked. 
The  14-section  booklet.  Quality  Control  Form  for  Monitored  Schools,  is  included  in  the  Report 
on  Data  Collection  Activities  for  the  1996  National  Assessment  of  Educational  Progress  (Westat, 
Inc.,  1996). 

If  four  or  more  students  were  absent  from  the  session,  a makeup  session  was  to  be  held. 

If  the  original  session  had  been  monitored,  the  makeup  session  was  also  monitored.  This  required 
coordination  of  scheduling  between  the  quality  control  monitor  and  assessment  administrator. 


4.3.5  Participation  of  Department  of  Defense  Education  Activity  Schools  (DoDEA) 

The  schools  run  by  the  Department  of  Defense  at  military  bases  and  other  installations 
around  the  world  participated  in  the  NAEP  State  Assessment  for  the  second  time  in  1996.  The 
participation  of  the  selected  schools  was  mandated  by  DoDEA.  To  accommodate  the  geographic 
diversity  of  DoDEA  schools,  some  minor  adaptations  were  made  in  the  preparatory  activities 
used  for  the  other  jurisdictions. 

The  data  collection  in  DoDEA  schools  was  expanded  in  1996  so  that  both  DDESS  and 
DoDDS  DoDEA  schools  were  surveyed.  In  1994,  only  the  schools  at  overseas  installations  were 
sampled  as  part  of  the  State  Assessment.  Also,  DoDEA  chose  to  conduct  science  assessments  at 
grade  4 (in  other  State  NAEP  schools,  science  was  conducted  only  at  grade  8)  so  that  both 
mathematics  and  science  data  were  collected  at  both  grades  4 and  8 in  DoDEA  schools. 
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Many  of  the  quality  control  monitors  hired  for  the  DoDEA  schools  were  based  overseas, 
and  many  had  previous  experience  working  within  the  DoDEA  system.  They  were  referred  to 
Westat  by  DoDEA.  All  quality  control  monitors  for  the  DoDEA  schools  attended  the  quality 
control  training  in  Los  Angeles  and  several  assessment  administrator  training  sessions  in  the 
geographic  areas  in  which  they  worked. 

The  samples  of  students  to  be  assessed  in  the  DoDEA  schools  were  selected  in  the 
Westat  home  office,  using  standard  NAEP  procedures,  from  lists  of  students  produced  in  the 
DoDEA  offices  in  Northern  Virginia.  Due  to  privacy  concerns,  only  student  ID  numbers  and  not 
student  names  appeared  on  the  DoDEA  lists.  Thus,  after  sampling,  the  Administration  Schedules 
contained  only  the  ID  numbers,  and  the  assessment  administrators  consulted  school  records  and 
added  the  names  of  the  students  to  the  Administration  Schedules  prior  to  the  assessments. 

Two  field  supervisors  were  hired  specifically  to  conduct  assessment  administrator 
trainings  and  monitor  quality  control  monitors  in  the  DoDDS  schools.  The  DoDEA  liaison  in 
Northern  Virginia,  who  essentially  functioned  as  the  state  coordinator,  arranged  the  assessment 
administrator  training  sessions,  all  of  which  were  held  in  schools  or  other  facilities  on  the  bases. 
In  many  cases,  the  quality  control  monitors  were  required  to  obtain  special  clearances  through 
DoDEA  to  visit  the  bases  for  training  and  the  assessments. 

The  assessments  in  DoDEA  schools  were  conducted  using  the  same  procedures  as  in  all 
State  Assessment  schools  with  the  one  exception  that  DoDEA  included  science  assessments  at 
both  grades  4 and  8. 


4.3.6  Exclusion  of  Students  from  the  Assessment 

Due  to  recent  interest  in  including  as  many  students  as  possible  in  NAEP  and  other 
educational  assessments,  efforts  were  initiated  in  the  1995  field  test  to  explore  the  impact  of 
redefining  the  NAEP  inclusion  criteria  for  students  with  disabilities  and/or  limited  English 
proficiency  (SD/LEP).  This  investigation  was  continued  in  1996  in  both  the  national  and  State 
Assessments. 

The  approach  taken  in  the  1996  State  Assessment  was  to  divide  the  school  sample  into 
two,  equal-size  subsamples,  referred  to  as  S 1 and  S2.  The  schools  in  the  SI  subsample  were 
asked  to  apply  the  “old”  (used  in  previous  years)  inclusion  criteria;  the  S2  schools  received  a 
“new,”  revised  criteria.  The  assessment  administration  for  a school  assured  that  the  appropriate 
set  of  inclusion  criteria  were  used  in  each  school.  Training  of  each  member  of  the  field  staff 
included  information  about  the  two  sets  of  inclusion  criteria.  Figures  4-1  and  4-2  describe  the 
criteria  for  S 1 and  S2. 
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Figure  4-1 

SI  Criteria 

A student  identified  on  the  Administration  Schedule  as  LEP  may  be  excluded  from  the 
assessment  if  he  or  she: 

1 . is  a native  speaker  of  a language  other  than  English, 

2.  has  been  enrolled  in  an  English-speaking  school  (not  including  a bilingual  education 
program)  for  less  than  two  years,  and 

3.  is  judged  to  be  incapable  of  taking  part  in  the  assessment. 

A student  identified  on  the  Administration  Schedule  as  SD  or  an  equivalent  classification 
may  be  excluded  from  the  assessment  if: 

1.  the  student  is  mainstreamed  less  than  50  percent  of  the  time  in  academic  subjects  and  is 
judged  incapable  of  participating  meaningfully  in  the  assessment,  or 

2.  the  Individualized  Education  Plan  (IEP)  team  or  equivalent  group  has  determined  that  the 
student  is  incapable  of  participating  meaningfully  in  the  assessment. 

SD/LEP  students  meeting  the  above  criteria  should  be  assessed  if,  in  the  judgment  of 
school  staff,  they  are  capable  of  taking  the  assessment. 


Figure  4-2 

S2  Criteria 

A student  who  is  identified  on  the  Administration  Schedule  as  LEP  and  who  is  a native 
speaker  of  a language  other  than  English  should  be  included  in  the  NAEP  assessment  unless: 

1 . the  student  has  received  mathematics,  science,  and  language  arts  instruction  primarily  in 
English  for  less  than  three  school  years,  including  the  current  year,  or 

2.  the  student  cannot  demonstrate  his  or  her  knowledge  of  mathematics  or  science  in  English 
without  an  accommodation  or  adaptation. 

A student  identified  on  the  Administration  Schedule  as  SD  or  an  equivalent  classification 
should  be  included  in  the  NAEP  assessment  unless: 

1.  the  IEP  team  or  equivalent  group  has  determined  that  the  student  cannot  participate  in 
assessments  such  as  NAEP, 

2.  the  student's  cognitive  functioning  is  so  severely  impaired  that  she  or  he  cannot  participate, 
or 

3.  the  student's  IEP  requires  that  the  student  be  tested  with  an  accommodation  or  adaptation 
and  the  student  cannot  demonstrate  his/her  knowledge  of  mathematics  or  science  without  that 
accommodation  or  adaptation. 

The  school  person  most  knowledgeable  about  each  student  classified  as  IEP  or  LEP 
should  complete  an  SD/LEP  Questionnaire  about  the  student. 


The  preliminary,  unweighted  proportion  of  students  in  the  S1/S2  subsampling  suggest 
that  applying  the  new  (S2)  or  old  (SI)  criteria  result  in  virtually  no  change  in  the  proportions  of 
students  excluded  from  the  NAEP  assessments  as  SD  or  LEP.  For  example,  in  grade  4 public 
schools,  in  both  subsamples,  about  5.6-5 .7  percent  of  the  students  were  excluded  as  SD  and 
about  1.6-1 .7  percent  were  consistently  excluded  as  LEP  students.  The  rates  are  slightly  lower 
for  grade  8 public-school  students — just  below  five  percent  for  SD  exclusions,  and  about  one 
percent  for  LEP — and  again  consistent  across  the  two  subsamples.  The  rates  for  nonpublic  , 
schools  were  lower  still,  that  is,  consistently  less  than  half  the  size  of  the  public-school  rates  and 
very  similar  across  the  SI  and  S2  subsamples. 


4.3.7  School  and  Student  Participation 

Table  4-2  shows  the  results  of  the  state  coordinators’  efforts  to  gain  the  cooperation  of 
the  selected  schools.  Overall,  for  the  1996  State  Assessment  in  mathematics,  4,573  public 
schools  and  470  nonpublic  schools  for  grade  4 participated.  For  grade  8,  3,913  public  schools 
and  455  nonpublic  schools  participated. 

Participation  results  for  students  in  the  1996  State  Assessment  in  mathematics  are  given 
in  Table  4-3.  Over  148,000  fourth-grade  students  and  over  136,500  eighth-grade  students  were 
sampled.  As  can  be  seen  from  the  table  the  original  sample,  which  was  selected  by  the  NAEP 
state  supervisors,  comprised  approximately  144,500  (or  98%)  of  the  total  number  of  students 
sampled  for  grade  4,  and  approximately  134,000  (or  98%)  of  the  total  number  of  students 
sampled  for  grade  8.  The  original  sample  size  was  increased  somewhat  after  the  supplemental 
samples  had  been  drawn  (from  students  newly  enrolled  since  the  creation  of  the  original  lists). 


Table  4-2 

School  Participation , 1996  State  Assessment  in  Mathematics 


Grade  4 Grade  8 

Public  Nonpublic  Public  Nonpublic 


Number  of  schools  in  original  sample 
Number  of  schools  not  eligible  (closed,  no  4th 

4,980 

869 

4,482 

1,068 

grade) 

88 

126 

128 

220 

Number  of  eligible  schools  in  original  sample 
Non-cooperating  (e.g.,  school,  district,  or 

4,892 

743 

4,354 

848 

state  refusal) 

492 

224 

552 

299 

Cooperating 

Number  of  substitutes  provided  for  non- 

4,400 

519 

3,802 

549 

cooperating  schools 

Number  of  participating  substitutes  for  non- 

450 

144 

403 

154 

cooperating  schools 

Total  number  of  schools  participating  (after 

173 

17 

111 

12 

substitution) 

4,573 

470 

3,913 

455 

66 


Table  4-3 

Student  Participation,  1996  State  Assessment  in  Mathematics 


Grade  4 Grade  8 


Public Nonpublic Public Nonpublic 


Number  sampled 

137,892 

10,370 

128,608 

8,291 

Original  sample 

134,525 

10,249 

125,688 

8,230 

Supplemental  sample 

3,367 

121 

2,920 

61 

Percent  increase  in  original  sample 
by  adding  supplemental  sample 

2.5% 

1.2% 

2.3% 

0.7% 

Number  (%)  of  originally  sampled 
students  withdrawn 

5,432  (4.0%) 

148(1.4%) 

5,669  (4.5%) 

104(1.3%) 

Number  of  students  excluded1 

9,649 

117 

7,488 

44 

Number  (%)  of  sampled  students 
identified  as  SD 

15,573  (11.3%) 

291  (2.8%) 

13,489(10.5%) 

155(1.9%) 

Number  (%)  of  sampled  students 
excluded  as  SD 

7.678  (5.6%) 

104(1.0%) 

6,223  (4.8%) 

33  (0.4%) 

Number  (%)  of  sampled  students 
identified  as  LEP 

4,917(3.6%) 

186(1.8%) 

2,866  (2.2%) 

43  (0.5%) 

Number  (%)  of  sampled  students 
excluded  as  LEP 

2,215(1.6%) 

18(0.2%) 

1,388(1.1%) 

11  (0.1%) 

Number  of  students  to  be  assessed 

122,811 

10,105 

115,451 

8,143 

Number  of  students  assessed 

116,583 

9,704 

105,073 

7,764 

Original  sessions 

115,744 

9,661 

103,571 

7,749 

Makeup  sessions 

839 

43 

1,502 

15 

Student  participation  rates 
Before  makeups 

.94.2% 

95.6% 

89.7% 

95.2% 

After  makeups 

94.9% 

96.0% 

91.0% 

95.3% 

1 To  be  excluded,  a student  had  to  be  designated  as  SD  or  LEP  and  judged  incapable  of  participating  in  the  assessment, 
A student  could  be  identified  as  both  SD  and  LEP.  resulting  in  this  number  being  less  than  the  sum  of  the  students 
excluded  as  SD  or  LEP. 


4.3.8  Results  of  the  Observations 

During  the  assessment  sessions,  the  quality  control  monitors  observed  whether  the 
assessment  environment  was  adequate  or  inadequate  based  on  factors  such  as  room  size,  seating 
arrangements,  noise  from  hallways  or  adjacent  rooms,  and  lighting.  (If  the  room  was  unsuitable, 
however,  the  quality  control  monitors  did  not  routinely  ask  the  assessment  administrator  to  make 
other  arrangements.)  Of  the  3,776  monitored  assessment  sessions  where  quality  control  monitors 
recorded  an  observation,  the  quality  control  monitors  felt  that  96  percent  of  the  sessions  were 
held  in  suitable  surroundings 

The  Manual  for  Assessment  Administrators  encouraged  assessment  administrators  to  use 
an  assistant  during  the  assessment  session,  a suggestion  that  came  from  the  earliest  state 
assessment  in  1990.  To  measure  how  frequently  that  advice  was  heeded,  quality  control  monitors 
noted  whether  an  assistant  was  used  in  the  monitored  sessions.  The  results  indicate  that  assistants 
were  used  for  60  to  70  percent  of  the  public-school  sessions,  with  the  largest  percentage 
(66-70%-)  noted  for  grade  8 sessions.  In  nonpublic  schools,  however,  an  assistant  was  employed 
less  often  (29-40%  of  t?»c  time),  which  is  possibly  a reflection  of  fewer  staff  resources  and 
generally  smaller  session  sizes  in  nonpublic  schools.  Assessment  administrators  used  assistants 
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in  varying  capacities.  The  Manual  for  Assessment  Administrators  was  very  emphatic  that  only  a 
NAEP-trained  person  could  actually  administer  the  assessment  session.  Almost  always,  assistants 
helped  to  supervise  the  session  and  *o  prepare,  distribute,  and  collect  assessment  materials  and/or 
booklets. 

The  assessment  administrators  were  asked  to  estimate  the  total  time  that  they  spent  on 
the  preparations  for  and  the  conduct  of  the  assessment,  including  their  attendance  at  the  training 
session.  Estimates  for  1996  were  similar  to  those  for  1992  because  two  subjects  were  assessed  in 
each  of  these  years  (compared  to  1994  when  only  one  subject  was  assessed).  In  1996,  a majority 
of  the  assessment  administrators  with  grade  4 sessions  (63%  in  public  and  82%  in  nonpublic 
schools)  stated  that  they  spent  less  than  20  hours  on  the  assessment.  For  grade  8,  however,  only 
30  percent  of  the  assessment  administrators  in  public  schools,  compared  to  73  percent  of  those  in 
nonpublic  schools,  spent  fewer  than  20  hours.  The  variation  in  time  distribution  for  grade  8 
public  schools,  particularly  compared  to  public  schools  at  grade  4,  is  most  likely  due  to  the  fact 
that  two  session  types  (mathematics  and  science)  were  usually  conducted  by  each  grade  8 
assessment  administrator,  but  only  one  session  type  (mathematics)  was  held  at  grade  4.  This  does 
not  appear  to  hold  true  for  nonpublic  schools,  however,  where  the  distribution  of  time  spent  is 
more  similar  for  grades  4 and  8.  It  is  evident  that  assessment  administrators  in  nonpublic  schools 
spent  fewer  hours  overall  on  the  assessment  than  did  assessment  administrators  in  public  schools. 
Potential  explanations  might  be  the  generally  smaller  sessions  sizes  in  nonpublic  schools  (i.e., 
fewer  materials  to  prepare  and  ship)  and  the  possibility  that  some  grade  8 schools  may  have  used 
more  than  one  assessment  administrator  with  each  assessment  administrator  conducting  one 
session  (but  compiling  a larger  total  time  for  all  sessions  combined). 

Quality  control  monitors  reported  that  they  observed  the  opening  of  assessment  booklet 
bundles  for  3,539  (or  89%)  of  the  monitored  sessions,  and  it  is  assumed  that  these  bundles  were 
opened  at  the  proper  time.  In  two  percent  of  the  sessions,  however,  the  bundle  opening  was  not 
observed  due  to  quality  control  monitor  error,  (e.g.,  the  quality  control  monitor  was  late,  in  the 
wrong  place,  or  miscommunicated  with  the  assessment  administrator);  presumably,  some  (or 
probably  most)  of  these  bundles  were  opened  at  the  correct  time.  For  another  two  percent  of 
sessions,  the  quality  control  monitors  were  unable  to  observe  the  bundle  opening  that  occurred 
early  due  to  assessment  administrator  error  (e.g.,  the  assessment  administrator  misunderstood  the 
procedures,  felt  more  time  was  needed,  had  scheduling  conflicts,  or  needed  to  prepare  for 
multiple  sessions  starting  at  the  same  time).  Information  on  the  opening  of  the  assessment 
booklet  bundles  was  not  reported  for  the  remaining  seven  percent  of  the  monitored  sessions. 

When  queried,  the  quality  control  monitors  felt  most  positive  about  the  attitudes  of  the 
assessment  administrators  and  somewhat  less  positive  about  the  attitudes  of  other  school  staff 
and  the  students  towards  the  assessment. 

Quality  control  monitors  concluded  the  summary  section  by  assigning  a final  rating  of 
the  assessment  administrator’s  performance.  With  this  rating,  the  quality  control  monitor 
reconsidered  the  session  from  the  vantage  point  of  how  well  it  would  have  gone  without  the 
quality  control  monitor’s  presence.  Eighty-four  percent  of  the  assessment  administrators  in 
monitored  sessions  were  self-reliant  or  needed  to  consult  the  quality  control  monitors  for  only 
one  or  two  minor  items.  Only  about  four  or  five  percent  had  serious  difficulty  conducting  the 
session  (that  is,  relied  on  the  quality  control  monitor  to  initiate  procedures  or  conduct  the 
session). 

After  the  conclusion  of  the  assessment  sessions,  Westat  mailed  state  coordinators  a short 
survey  to  obtain  their  reactions  to  the  operations  associated  with  the  1996  State  Assessment  and 


any  suggestions  they  had  for  improving  the  program.  Thirty-seven  state  coordinators  responded 
by  returning  the  survey  or  by  providing  their  responses  over  the  telephone.  A detailed  summary 
of  the  state  coordinators’  responses  is  contained  in  the  Report  on  Data  Collection  Activities  for 
All  States  (Westat,  Inc.,  1996),  which  was  distributed  to  state  coordinators  in  October  1996. 
Some  of  the  responses  from  the  state  coordinators  included: 

• Fifteen  of  the  37  reporting  jurisdictions  mandated  participation  in  the  1996 
State  Assessment; 

• Seven  jurisdictions  reported  that  they  helped  gain  the  cooperation  of 
nonpublic  schools.  Most  of  these  provided  a letter  from  the  state 
superintendent  of  schools,  and  others  answered  questions. 

• Twenty-nine  jurisdictions  used  the  computer  system  throughout  the  Field 
testing  period.  Seven  jurisdictions  used  the  system  initially  but  not 
necessarily  during  the  assessment  period,  and  one  jurisdiction  did  not  use  the 
system  at  all.  The  jurisdictions  seemed  to  be  comfortable  with  the  computer 
system  and  were  able  to  use  it  effectively.  A fairly  common  suggestion  was 
to  expand  the  documentation  and  capabilities  regarding  label  production. 

• Of  the  jurisdictions  reporting  on  staff  time  devoted  to  NAEP,  state 
coordinators  spent  an  average  of  34  days  (ranging  from  2 to  100  days)  on 
NAEP  activities,  and  other  staff  spent  an  average  of  28  days  (ranging  from  2 
to  85  days). 

• Reactions  to  the  1996  State  Assessment  were  quite  positive.  Twenty-five  of 
th ; 28  state  coordinators  who  expressed  an  opinion  said  that  the  assessments 
went  “very  well”  or  “well”— even  though  this  was  a challenging  year  in 
terms  of  bad  weather,  missed  instruction  time,  and  school  staff  burden. 
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Chapter  5 


PROCESSING  AND  SCORING  ASSESSMENT  MATERIALS1 

Patrick  B.  Bourgeacq,  Charles  L Brungardt,  Luann  Forinash,  Mary  Lynn  Helscher, 
Tillie  Kennel,  Linda  L.  Reynolds,  Tim  Robinson,  Mary  Schulte, 

Connie  Smith,  Patricia  M.  Stearns,  and  Bradley  J.  Thayer 
National  Computer  Systems 


5.1  OVERVIEW 

This  chapter  reviews  the  processing  and  scoring  activities  conducted  by  National  Computer 
Systems  (NCS)  for  the  1996  NAEP  State  Assessment.  The  1996  assessment  presented  the  greatest 
challenge  in  processing  and  scoring  NAEP  data  to  date.  For  this  assessment.  NCS  was  charged  with 
processing  and  scoring  the  largest  assessment  in  the  history  of  NAEP  in  the  shortest  amount  of  time. 
Further,  image  scanning  processes,  eliminating  almost  all  paper  handling  during  scoring  and  improving 
monitoring  and  reliability  scoring,  increased  to  nearly  twice  that  of  the  1994  assessment.  In  the  early 
1990s,  NCS  developed  and  implemented  flexible,  innovatively  designed  processing  programs  and  a 
sophisticated  Process  Control  System  that  allows  the  integration  of  data  entry  and  workflow  management 
systems  to  accomplish  this  work. 

This  chapter  begins  with  a description  of  the  various  tasks  performed  by  NCS.  detailing  printing, 
distribution,  receipt  control,  scoring,  and  processing  activities.  It  also  discusses  specific  activities 
involved  in  processing  the  assessment  materials,  and  presents  an  analysis  of  several  of  those  activities. 
The  chapter  provides  documentation  for  the  professional  scoring  effort — scoring  guides,  training  papers 
papers  illustrating  sample  score  points,  calibration  papers,  calibration  bridges,  and  interreader  reliability 
reports.  The  detailed  processing  specifications  and  documentation  of  the  NAEP  Process  Control  S>  stem 
are  presented  in  the  final  sections  of  the  chapter. 


5.1.1  Innovations  for  1996 

Much  of  the  information  necessary  for  documentation  of  accurate  sampling  and  for  calculating 
sampling  weights  is  collected  on  the  Administration  Schedules  that,  until  1993,  were  painstakingly  filled 
out  by  hand  by  Westat  administrative  personnel.  In  1994,  for  the  first  time,  much  of  the  work  was 
computerized — book.ets  were  preassigned  and  booklet  ID  numbers  were  preprinted  on  the 
Administration  Schedule.  When  Westat  personnel  received  the  documents,  they  filled  in  only  the 
“exception”  information.  This  new  method  also  permitted  computerized  updating  of  information  when 
the  Administration  Schedules  were  received  at  NCS,  eliminating  the  need  to  sort  and  track  thousands  ol 
pieces  of  paper  through  the  processing  stream. 

The  introduction  of  image  processing  and  image  scoring  further  enhanced  the  work  ot  NAEP. 
Image  processing  and  scoring  were  successfully  piloted  in  a side-by-side  study  conducted  during  the 
1993  NAEP  field  test,  and  so  became  the  primary  processing  and  scoring  methods  lor  the  1994  and  1996 


' Patrick  Bourgeacq  is  the  project  director  for  scoring.  All  of  the  authors  were  involved  m the  piin.essing  and  '.oiing  pr<  ■.  odinc- 
for  the  NAEP  State  Assessments.  Jeff  Haberstroh  and  Chances  Jones  of  Educational  Testing  Service  contributed  to  the 
professional  scoring  section  of  this  chapter. 
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State  Assessments.  Image  processing  allowed  the  automatic  collection  of  handwritten  demographic  data 
from  the  administrative  schedules  and  the  student  test  booklet  covers  through  intelligent  character 
recognition  (ICR).  This  service  was  a benefit  to  the  jurisdictions  participating  in  NAEP  because  they 
were  able  to  write  rather  than  grid  certain  information — a reduction  of  burden  on  the  schools.  Image 
processing  also  made  image  scoring  possible,  eliminating  much  of  the  time  spent  moving  paper  as  part  of 
the  scoring  process.  The  images  of  student  responses  to  be  scored  were  transmitted  electronically  to  the 
scoring  center,  located  at  a separate  facility  from  where  the  materials  were  processed.  This  process 
enhanced  the  reliability  and  monitoring  of  scoring  and  allowed  both  NCS  and  ETS  to  focus  attention  on 
the  intellectual  process  of  scoring  student  responses. 

Tables  5-1  and  5-2  give  an  overview  of  the  processing  volume  and  the  schedule  for  the  1996 
NAEP  State  Assessment. 


Table  5-1 

1996  NAEP  State  Assessment  Processing  Totals 


Document/Category 

Totals 

Number  of  sessions 

15,487 

Assessed  student  booklets 

356,447 

Absent  student  booklets 

27,743 

Excluded  student  booklets 

25,713 

SD/LEP  questionnaires 

47,708 

School  questionnaires 

9,470 

Teacher  questionnaires 

39,311 

Scanned  documents 

356,447 

Scanned  sheets 

9,829,970 

Key-entered  documents1 

0 

'No  Braille  booklets  and  only  one  large-print  booklet  were  received  from 
the  1995  field  test.  Rather  than  key  enter  only  one  booklet,  the  decision 
was  made  to  bypass  the  key-entry  stage  and  let  the  scoring  center  score  it 
directly  from  the  booklet.  Thus,  there  were  zero  kev-cntcrcd  documents. 


Table  5-2 

1996  NAEP  State  Assessment,  NCS  Schedule 


Activity 

Planned 
Start  Date 

Planned 
Finish  Date 

Actual 
Start  Date 

Actual 
Finish  Date 

Printing 

Grade  4 Mathematics  Teacher 

9/2/95 

12/11/95 

9/2/95 

12/11/95 

Questionnaire  Roster  delivered  to  NCS 
Grade  8 Teacher  Questionnaires 

10/12/95 

10/12/95 

10/16/95 

10/16/95 

delivered  to  NCS 

Administration  Schedule  delivered  to 

10/12/95 

10/12/95 

10/16/95 

10/16/95 

NCS 

Grade  8 School  Characteristics  and 

10/18/95 

10/18/95 

10/23/95 

10/23/95 

Policies  Questionnaires  at  NCS 

10/20/95 

10/20/95 

10/23/95 

10/23/95 

SD/LEP  Roster  delivered  to  NCS 
Grade  4 School  Characteristics  and 

10/20/95 

10/20/95 

10/24/95 

10/24/95 

Policies  Questionnaires  at  NCS 
Grade  8 Mathematics  spiral  material  at 

10/20/95 

10/20/95 

10/25/95 

10/25/95 

NCS 

10/23/95 

1 1/2/95 

10/18/95 

1 1/3/95 

Pre-packaging  begins 
Grade  4 mathematics  Teacher 

10/23/95 

12/20/95 

10/16/95 

12/1/95 

Questionnaires 

Grade  4 mathematics  spiral  material  at 

10/26/95 

10/26/95 

11/1/95 

11/1/95 

NCS 

Grade  8 mathematics  Teacher 

10/26/95 

11/1/95 

11/1/95 

1 1/1/95 

Questionnaires  at  NCS 
NCS/ETS  meet  to  review  items  and 

10/30/95 

10/30/95 

10/25/95 

10/25/95 

scoring  schedule 

1 1/2/95 

11/3/95 

11/2/95 

1 1/3/95 

State  supervisor  training 
Administration  Schedule  address  file 

1 1/9/95 

11/12/95 

11/9/95 

11/11/95 

from  Westat 

95%  session  data  file  of  schools  from 

1 1/20/95 

1 1/20/95 

1 1/22/95 

1 1/22/95 

Westat 

1 1/22/95 

1 1/22/95 

11/22/95 

1/5/96 

SD/LEP  Questionnaire  delivered  to  NCS 

1 1/22/95 

1 1/22/95 

12/5/95 

12/11/95 

Print  Administration  Schedules 

Ship  Administration  Schedules  to  Westat 

1 1/27/95 

11/27/95 

1 1/28/95 

10/23/95 

state  supervisors 

1 1/29/95 

11/29/95 

10/23/95 

10/23/95 

All  materials  at  NCS  for  packaging 
State  supervisor  training  materials 

1 1/29/95 

12/1/95 

12/1/95 

12/15/95 

shipped 

12/15/95 

12/15/95 

12/13/95 

12/13/95 

School  address  file  from  Westat 

12/18/95 

12/18/95 

11/29/95 

1 1/29/95 

Final  packaging 

12/26/95 

2/3/96 

12/26/95 

2/7/96 

Receiving 

1/30/96 

3/5/96 

2/6/96 

3/12/96 

Processing 

2/2/96 

3/22/96 

2/6/96 

4/5/96 

PSC  selects  mathematics  table  leaders 

3/1/96 

3/1/96 

2/1/96 

2/28/96 

Scoring  training  preparation 

3/4/96 

3/22/96 

3/4/96 

3/22/96 

Scorers  assigned  to  teams 

3/11/96 

3/11/96 

3/11/96 

3/11/96 

Training  and  scoring 

3/13/96 

5/3/96 

3/13/96 

5/6/96 
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Table  5-2  (continued) 

1996  NAEP  State  Assessment,  NCS  Schedule 


Activity 

Planned 
Start  Date 

Planned 
Finish  Date 

Actual 
Start  Date 

Actual 
Finish  Date 

Weights  data  shipped  - grade  4 

3/15/96 

3/18/96 

3/28/96 

4/22/96 

Weights  data  shipped  - grade  8 

3/30/96 

4/1/96 

4/8/96 

4/22/96 

Grade  8 mathematics  weights 

5/4/96 

5/6/96 

5/3/96 

5/3/96 

Grade  4 mathematics  weights 
School  Characteristics  and  Policies 

5/4/96 

5/6/96 

5/9/96 

5/9/96 

Questionnaires  data  tape  shipped  to  ETS 
Teacher  Questionnaires  data  tape 

7/1 1/96 

7/12/96 

7/1 1/96 

7/1 1/96 

shipped 

SD/LEP  Questionnaires  data  shipped  to 

7/18/96 

7/19/96 

7/24/96 

7/24/96 

ETS 

7/26/96 

7/29/96 

8/7/96 

8/7/96 

5.2  PRINTING 
5.2.1  Overview 

For  the  1996  NAEP  assessments,  255  unique  documents  were  designed.  NCS  printed  more  than 
1 ,900,000  booklets  and  forms,  totaling  over  58  million  pages. 

Printing  preparations  began  with  the  design  of  the  booklet  covers  in  June  1995.  This  was  a 
collaborative  effort  involving  staff  from  ETS,  Westat,  and  NCS.  Because  the  goal  was  to  design  one 
format  for  use  with  all  of  the  booklets,  necessary  data  elements  to  be  collected  for  the  different 
assessment  types  had  to  be  agreed  upon.  In  a similar  collaboration  with  ETS  and  Westat,  NCS  prepared 
Administration  Schedules  and  questionnaire  rosters,  and  the  camera-ready  copies  for  the  documents  were 
created  and  edited.  The  printing  of  assessment  booklets,  questionnaires,  and  tracking  forms  for  the  main 
and  state  assessments  was  complete  by  December  1 1,  1995. 


5.2.2  State  Assessment  Printing 

The  printing  effort  for  the  State  Assessment  materials  began  with  the  receipt  of  camera-ready 
copy  of  short-term  trend  cognitive  mathematics  blocks.  Camera-ready  data  for  the  new  mathematics 
blocks  were  created  by  ETS,  as  were  some  of  the  directions  and  all  of  the  background  blocks. 

Because  large  numbers  of  documents  were  to  be  printed  in  a relatively  short  period  of  time, 
preliminary  composition  work  was  begun  by  the  NCS  printer  in  Columbia,  Pennsylvania,  and  the 
required  numbers  of  negatives  for  each  booklet  component  were  made.  Performing  these  preliminary- 
tasks  was  crucial  to  meeting  the  delivery  schedule. 

The  actual  assembly  of  booklets  began  after  all  parts  needed  for  a particular  booklet  were 
received  and  the  Office  of  Management  and  Budget  (OMB)  had  given  its  approval  to  print.  ETS  supplied 
booklet  maps  that  specified  the  order  of  blocks  in  each  booklet  (see  Chapter  2,  Table  2-4,  for  the  contents 
of  each  booklet).  Using  these  booklet  maps  and  mock-ups  of  booklets  as  guides,  the  NCS  printer 


assembled  prepared  negatives  into  complete  booklets.  Generally,  five  weeks  elapsed  between  receipt  of 
final  copy  and  delivery  of  printed  booklets. 

The  printer  forwarded  proofs  of  the  booklets  and  questionnaires  to  ETS  and  to  NCS  for  review 
and  approval  to  print.  Clean-up  work  and  changes,  where  necessary,  were  indicated  on  the  proofs,  which 
were  returned  to  the  printer.  Once  approved,  the  booklets  were  printed. 

As  the  booklets  and  forms  were  printed,  pallets  of  documents  were  received  and  entered  into 
NCS’s  Inventory  Control  system.  Sample  booklets  were  selected  and  quality-checked  for  printing  and 
collating  errors.  All  printing  for  the  1996  NAEP  State  Assessment  in  mathematics  was  completed  by 
December  11,  1995. 
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PACKAGING  AND  SHIPPING 


53.1  Distribution 

The  distribution  effort  for  the  1996  NAEP  State  Assessment  involved  packaging  and  mailing 
documents  and  associated  forms  and  materials  to  individual  schools.  The  NAEP  Materials  Distribution 
System,  initially  developed  by  NCS  in  1990  to  control  shipments  to  the  schools  and  supervisors,  was 
utilized  again  in  1996.  Files  in  the  system  contained  the  names  and  addresses  for  shipment  of  materials, 
scheduled  assessment  dates,  and  a listing  of  all  materials  available  for  use  by  a participant.  Changes  to 
any  of  this  information  were  made  directly  in  the  distribution  file  either  manually  or  via  file  updates 
provided  by  Westat.  Figure  5-1  illustrates  the  process  flow  for  the  accountability  system  and  online 
bundle  assignment  and  distribution  system  utilized  for  NAEP. 

Bar  code  technology  continued  to  be  utilized  in  document  control.  To  identify  each  document, 
NCS  utilized  a unique  ten-digit  numbering  system.  This  numbering  system  consisted  of  the  three-digit 
booklet  number  or  form  type,  a six-digit  sequential  number,  and  a check  digit.  Each  form  was  assigned  a 
range  of  ID  numbers.  Bar  codes  reflecting  this  ID  number  were  applied  to  the  front  cover  of  each 
document  by  NCS  bar  code  processes  and  high-speed  inkjet  printers. 

Once  all  booklets  from  a subject  area  were  bar  coded,  they  were  spiraled  and  bundled  into  groups 
of  eleven  documents.  For  State  Assessment  samples  in  mathematics,  NCS  spiraled  the  booklets  according 
to  the  pattern  dictated  by  ETS  in  the  bundle  maps.  Booklets  were  spiraled  in  such  a manner  that  each 
booklet  appeared  in  the  first  position  in  a bundle  approximately  the  same  number  of  times  and  that  the 
booklets  were  evenly  distributed  across  the  bundles.  This  assured  that  sample  sizes  of  individual  booklet 
types  would  not  be  jeopardized  if  entire  bundles  were  not  used. 

All  booklets  had  to  be  arranged  in  the  exact  order  listed  on  the  bundle  header  sheet.  To  ensure 
the  accuracy  of  each  bundle  and  the  security  of  the  NAEP  assessment,  a quality  control  plan  was  utilized 
to  verify  the  document  order  of  each  bundle  and  to  account  for  all  booklets.  All  bundles  that  contained  a 
bundle  slip  were  taken  to  a bar  code  reader/document  transport  machine  where  they  were  scanned  to 
interpret  ecch  bundle’s  ' ar  code.  The  file  of  scanned  bar  codes  was  then  transferred  from  the  personal 
computer  connected  to  the  scanner  to  a mainframe  data  set. 

The  unique  bundle  number  on  the  header  sheet  informed  the  system  program  what  type  of  bundle 
should  follow.  A computer  job  was  run  to  compare  the  bundle  type  expected  to  the  sequence  of  booklets 
that  was  scanned  after  the  header.  This  job  also  verified  that  the  appropriate  number  of  booklets  was 
included  in  each  bundle.  Any  discrepancies  were  printed  on  an  error  listing.  The  NCS  packaging 
department  corrected  the  error  and  the  bundle  was  again  read  into  the  system.  This  process  was  repeated 
until  no  discrepancies  existed.  By  using  this  quality-control  plan,  NCS  could  verify  the  document  order 
of  each  bundle  and  account  for  all  booklets. 

Once  a bundle  cleared  the  bundle  quality  control  process,  it  was  shrink-wrapped  and  flagged  on 
the  system  as  ready  for  distribution.  In  the  State  Assessment,  the  bundles  were  not  to  be  opened  until  45 
minutes  before  the  assessment.  The  mathematics  bundles  were  shrink-wrapped,  strapped,  and  a label  was 
placed  on  the  lop  of  each  bundle  that  read  “Do  Not  Open  Until  45  Minutes  Before  Assessment.” 
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Figure  5-1 

1996  NAEP  State  Assessment  Materials  Distribution  Flow  Chart 


BEST  COpY  AVAILABLE 


Once  all  bundles  for  a subject  area  passed  the  bundle  quality  control  process, 
information  from  the  bundle  quality  control  file  was  uploaded  to  the  mainframe  computer  system 
and  used  in  the  creation  of  Administration  Schedules.  All  Administration  Schedules  for  each 
scheduled  session  were  pre-printed  with  the  booklet  IDs  designated  for  that  session.  Three 
bundles  of  booklets  were  pre-assigned  to  each  session,  giving  each  session  33  booklets.  This 
number  most  closely  approximated  the  average  projected  session  size  plus  an  additional  supply 
of  booklets  for  any  extra  students. 

Using  sampling  files  provided  by  Westat,  NCS  assigned  bundles  to  schools  and 
customized  the  packing  lists.  File  data  from  Westat  was  coupled  with  the  file  of  bundle  numbers 
and  the  corresponding  booklet  numbers.  This  file  was  then  used  to  pre-print  all  booklet 
identification  numbers,  school  name,  school  number  and  session  type,  directly  onto  the  scannable 
Administration  Schedule.  As  a result,  every  pre-scheduled  session  had  specific  bundles  assigned 
to  it  in  advance.  This  increased  the  quality  level  of  the  booklet  accountability  system  by  enabling 
NCS  to  identify  where  any  booklet  should  be  at  any  time  during  the  assessments.  It  also 
eliminated  the  possibility  of  transcription  errors  by  assessment  administrators  for  booklet  ID 
numbers.  Lastly,  by  pre-printing  booklet  ID  numbers,  the  burden  on  the  schools  for  transcription 
of  data  was  notably  reduced.  NCS  distributed  the  pre-printed  Administration  Schedules  to  state 
supervisors.  The  supervisors  subsequently  forwarded  them  to  the  assessment  administrators  in 
the  schools  before  their  session  materials  arrived.  Having  the  preprinted  Administration 
Schedules  early  assisted  with  sampling  in  the  schools. 

Distribution  of  materials  for  the  State  Assessment  was  accomplished  in  five  waves  or 
shipment  dates.  Except  for  wave  "zero,"  session  materials  were  sent  to  a school  two  weeks 
before  the  assessment  date.  All  school  materials  were  sent  directly  to  an  assessment 
administrator  at  a school  or  school  district.  Materials  for  Alaska,  Guam.  Hawaii  and  Department 
of  Defense  Education  Activity  (DoDEA)  schools  were  sent  first  in  wave  "zero."  These  shipments 
required  using  an  alternate  carrier  to  ensure  timely  delivery  and  minimize  the  impact  of  customs 
delays.  NCS  received  customs  forms  provided  by  the  carrier.  These  forms  were  attached  to  the 
outside  of  the  shipment  boxes.  Information  such  as  address,  school  number  and  return  address 
were  pre-printed  on  these  forms.  Extra  forms  were  also  sent  for  returning  boxes  back  to  NCS  in 
Iowa  City.  The  remaining  four  waves  were  sent  out  weekly  based  on  the  schools  scheduled 
assessment  date.  In  case  any  of  the  quantities  were  insufficient  for  the  assessment,  administrators 
were  given  the  NAEP  toll-free  number  to  request  additional  materials. 

Initially,  a total  of  9,930  sets  of  session  materials  were  shipped  for  the  1996  State 
Assessments.  Approximately  3,000  additional  shipments  of  booklets  and  miscellaneous  materials 
were  sent.  All  outbound  shipments  were  recorded  in  the  NCS  outbound  mail  management 
system.  This  was  accomplished  by  having  a bar  code  containing  the  school  number  on  each 
address  label.  This  bar  code  was  read  into  the  system,  which  determined  the  routing  of  the 
shipment  and  the  charges.  Information  was  recorded  in  a file  on  the  system  that,  at  the  end  of 
each  day,  was  transferred  by  a PC  upload  to  the  mainframe.  A computer  program  could  then 
access  information  to  produce  reports  on  all  shipments  sent,  regardless  of  the  carrier  used.  These 
reports  helped  NCS  phone  staff  trace  shipments  for  state  supervisors  and  assessment 
administrators. 
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5.3.2  Short  Shipment  and  Phones 


A toll-free  telephone  line  was  maintained  for  school  administrators  to  request  additional 
materials  for  the  State  Assessments.  To  process  a shipment,  NCS  phone  staff  asked  the  caller  for 
information  such  as  PSU,  school  ID,  assessment  type,  city,  state,  and  zip  code.  This  information 
was  then  entered  into  the  online  short  shipment  system  and  the  school’s  mailing  address  would 
be  displayed  on  the  screen  to  verify  with  the  caller.  The  system  allowed  NCS  staff  to  change  the 
shipping  address  for  individual  requests.  The  clerk  proceeded  to  the  next  screen  that  displayed 
the  materials  to  be  selected.  After  the  requested  items,  due  date  and  method  of  shipment  were 
entered,  the  system  produced  a packing  list  and  mailing  labels.  Phone  staff  also  took  phone  calls 
concerning  initial  shipment  delivery  dates,  tracing  a shipment,  and  questions  concerning  NAEP. 
Approximately  3,750  calls  were  received  regarding  the  1996  NAEP  State  Assessments.  Table  5-4 
lists  the  types  of  requests  and  number  of  calls  per  request. 


Table  5-4 

1996  NAEP  State  Assessment 
Phone  Request  Summary 


Number  of  Calls 

Request 

46 

Additional  test  booklets — increase  in  session 

977 

Additional  SD/LEP  questionnaires 

940 

Additional  teacher  questionnaires 

515 

Miscellaneous  materials  (excluding  science  kits) 

212 

Science  kits 

248 

Missing  materials  in  shipments 

51 

Add  on  school 

236 

Tracing  Shipments 

400 

Other  (delivery  dates,  NAEP  questions) 

5.4  PROCESSING 
5.4.1  Overview 

The  following  describes  the  various  stages  of  work  involved  in  receiving  and  processing 
the  documents  used  in  the  1996  NAEP  State  Assessment.  NCS  staff  created  a set  of 
predetermined  rules  and  specifications  for  the  processing  departments  within  NCS  to  follow. 
Project  staff  performed  a variety  of  procedures  on  materials  received  from  the  assessment 
administrators  before  releasing  these  materials  into  the  NCS  NAEP  processing  system.  Control 
systems  were  used  to  monitor  all  NAEP  materials  returned  from  the  field.  The  NAEP  Process 
Control  System  contained  the  status  of  sampled  schools  for  all  sessions  and  their  scheduled 
assessment  dates.  As  materials  were  returned,  the  Process  Control  System  was  updated  to 
indicate  receipt  dates,  to  record  counts  of  materials  returned,  and  to  document  any  problems 
discovered  in  the  shipments.  As  documents  were  processed,  the  system  was  updated  to  reflect 
processed  counts.  NCS  report  programs  were  utilized  to  allow  ETS,  Westat,  and  NCS  staff  to 
monitor  the  progress  in  the  receipt  control  operations.  The  processing  flow  is  illustrated  in  Figure 
5-2. 
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Figure  5-2 

1 996  NAEP  State  Assessment  Materials  Processing  Flow  Chart 


An  “alert”  process  was  used  to  record,  monitor,  and  categorize  all  discrepant  or 
problematic  situations.  Throughout  the  processing  cycle,  alert  situations  were  either  flagged  by 
computer  programs  or  identified  during  clerical  check-in  procedures.  Certain  alerts,  such  as 
missing  demographic  information  on  the  Administration  Schedule,  were  resolved  by  opening 
staff  retrieving  the  information  from  booklet  covers.  Alert  situations  that  could  not  be  resolved 
by  opening  personnel  were  described  on  alert  forms  that  were  forwarded  to  project  personnel  for 
resolution.  Once  resolved,  the  problems  and  resolutions  were  recorded  online  in  the  Process 
Control  System. 

NCS's  Workflow  Management  System  was  used  to  track  batches  of  student  booklets 
through  each  processing  step,  allowing  project  staff  to  monitor  the  status  of  all  work  in  progress. 
It  was  also  used  by  NCS  to  analyze  the  current  work  load,  by  project,  across  all  work  stations.  By 
routinely  monitoring  these  data,  NCS’s  management  staff  was  able  to  assign  priorities  to  various 
components  of  the  work  and  to  monitor  all  phases  of  the  data  receipt  and  processing. 


5.4.2  Document  Receipt  and  Tracking 

All  shipments  were  to  be  returned  to  NCS  packaged  in  their  original  boxes.  As 
mentioned  earlier,  NCS  packaging  staff  applied  a bar  code  label  to  each  box  indicating  the  NAEP 
school  ID  number.  When  a shipment  arrived  at  the  NCS  dock  area,  this  bar  code  was  scanned  to 
a personal  computer  file,  and  the  shipment  was  forwarded  to  the  receiving  area.  The  personal 
computer  file  was  then  transferred  to  the  mainframe  and  the  shipment  receipt  date  was  applied  to 
the  appropriate  school  within  the  Process  Control  System,  providing  the  status  of  receipts 
regardless  of  any  processing  delays.  Each  receipt  was  reflected  on  the  Process  Control  System 
status  report  provided  to  the  NCS  receiving  department  and  supplied  to  Westat  via  electronic  tile 
transfer  and  in  hard-copy  format.  ETS  also  received  a hard  copy.  The  Process  Control  System 
tile  could  be  manually  updated  to  reflect  changes,  if  necessary. 


Receiving  personnel  also  checked  the  shipment  to  verify  that  the  contents  of  the  box 
matched  the  school  and  session  indicated  on  the  label.  Each  shipment  was  checked  for 
completeness  and  accuracy.  Any  shipment  not  received  within  two  days  of  the  scheduled 
assessment  date  was  flagged  in  the  Process  Control  System  and  annotated  on  the  Process  Control 
System  report.  The  administration  status  of  these  delayed  shipments  was  checked  and  in  some 
cases  a trace  was  initiated  on  the  shipment. 

A new  requirement  for  NCS  was  to  open  all  shipments  within  48  hours  of  their  receipt 
and  to  key-enter  preliminary  processing  information  into  the  Process  Control  System  irom  the 
Administration  Schedule.  The  preliminary  information  was  written  on  the  Administration 
Schedule  by  assessment  administrators  and  consisted  ot  the  following: 


• School  number 

• Session  number 

• Original  test  date 

• Total  number  to  be  assessed 

• Total  number  assessed 
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This  preliminary  information,  used  to  provide  Westat  with  timely  student  response  rates, 
was  updated  with  actual  data  when  materials  passed  through  processing  error  free.  A 
completeness  flag  was  also  applied  to  the  process  control  file  by  NCS  opening  staff  if  any  part  of 
the  shipment  was  missing. 

If  multiple  sessions  were  returned  in  one  box,  the  contents  of  the  package  were  separated 
by  session.  The  shipment  was  checked  to  verify  that  all  booklets  preprinted  or  handwritten  on  the 
Administration  Schedule  were  returned  with  the  shipment  and  that  all  administration  codes  from 
booklet  covers  matched  the  Administration  Schedule.  If  discrepancies  were  discovered  at  any 
step  in  this  process,  the  receiving  staff  issued  an  alert  to  facilitate  tracking. 

If  the  administrator  indicated  that  a make-up  session  was  being  held  the  documents  were 
placed  on  holding  carts  until  the  make-up  session  documents  arrived.  If  no  make-up  session  was 
indicated,  Westat  was  contacted  for  the  status  of  the  missing  materials.  If  the  missing  materials 
were  to  be  returned,  the  documents  already  received  were  held  until  that  time.  If  the  materials 
were  not  being  returned,  processing  continued  and  the  appropriate  administration  code  was 
applied  to  the  Administration  Schedule. 

Once  all  booklets  listed  on  the  Administration  Schedule  for  a session  were  verified  as 
present,  the  entire  session  (both  the  Administration  Schedule  and  booklets)  was  batched  by  grade 
level  and  session  type.  Each  batch  was  assigned  a unique  batch  number.  This  number,  created  on 
the  Image  Capture  Environment  system  for  all  image-scannable  documents  and  on  the  Workflow 
Management  System  for  all  key-entry  and  OMR-scannable  documents,  facilitated  the  internal 
tracking  of  the  batches  and  allowed  departmental  resource  planning.  All  other  scannable 
documents  (School  Characteristics  and  Policies  Questionnaires,  Teacher  Questionnaires, 

SD/LEP  Questionnaires,  and  rosters)  were  batched  by  document  type  in  the  same  manner. 

Because  the  State  Assessment  mathematics  booklets  were  image-scannable,  batch 
numbers  for  these  documents  were  created  on  the  Image  Capture  Environment  system.  Sessions 
were  sorted  by  grade  level  and  automatically  uploaded  to  the  Workflow  Management  System 
after  batch  creation.  The  Administration  Schedule  for  these  document  types  was  used  as  a 
session  header  within  a batch. 

When  batching  State  Assessment  documents,  NCS  needed  to  allow  for  having  both 
image-scannable  and  key-entry  documents  present  in  the  same  session,  or  having  booklets  listed 
on  the  Administration  Schedule  that  would  not  be  present  in  processing.  This  was  due  to  the 
testing  accommodations  of  large-print  and  Braille  that  were  key-entry  documents.  Large-print 
booklets  had  to  be  processed  separately  from  the  Administration  Schedule  and  scannable 
booklets  in  their  session.  A key-entry  session  header  was  created  for  these  booklets.  This  session 
header  contained  the  school  ID  number  and  session  code  from  the  Administration  Schedule. 

The  1996  NAEP  State  Assessment  utilized  one  roster  to  document  and  track  the  School 
Characteristics  and  Policies  Questionnaire  and  the  Students  with  Disabilities/Limited  English 
Proficiency  (SD/LEP)  Questionnaire.  In  addition,  the  State  Assessment  used  the  Teacher 
Questionnaire  Roster  to  record  the  distribution  and  return  of  Teacher  Questionnaires. 

Some  questionnaires  may  not  have  been  available  for  return  with  the  shipment.  These 
were  returned  to  NCS  at  a later  date  in  an  envelope  provided  for  that  purpose.  The  questionnaires 
were  submitted  for  scanning  as  sufficient  quantities  became  available  for  batching. 
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Receipt  of  the  questionnaires  was  entered  into  the  system  using  the  same  process  as  was 
used  for  the  Administration  Schedule  described  in  previous  sections.  The  rosters  were  grouped 
with  other  rosters  of  the  same  type  from  other  sessions,  and  a batch  was  created  on  the  Image 
Capture  Environment  system.  The  batch  was  then  forwarded  to  scanning  where  all  information 
on  the  rosters  was  scanned  into  the  system. 

In  the  1996  NAEP  State  Assessment,  NCS  used  a sophisticated  booklet  accountability 
system  to  track  all  distributed  booklets.  As  stated  earlier,  prior  to  the  distribution  of  NAEP 
materials,  unique  booklet  numbers  were  read  by  bundle  into  a file.  Specific  bundles  were  then 
assigned  to  particular  supervisors  or  schools.  This  assignment  was  recorded  in  the  NAEP 
Materials  Distribution  System.  When  shipments  arrived  at  NCS  from  the  field,  all  used  booklets 
were  submitted  for  processing  and  a “processed  documents”  file  was  maintained.  Unused 
booklets  were  submitted  for  security  scanning  where  booklet  ID  bar  codes  were  read  and 
recorded  into  a separate  file.  This  file  and  the  “processed  documents”  file  were  later  compared  to 
the  original  bundle  security  file  for  individual  booklet  matching.  A list  of  unmatched  booklet  IDs 
was  printed  in  a report  used  to  confirm  non-receipt  of  individual  booklets.  Efforts  w'ere  made  to 
be  sure  unused  materials  from  the  State  Assessment  were  returned  by  school  personnel.  The  used 
hut  returned  booklet  IDs  were  also  read  by  the  bar  code  scanner  and  added  to  the  bundle  security 
file.  All  unused  materials  received  were  then  inventoried  and  sent  to  the  NCS  warehouse  for 
storage  while  awaiting  authorization  from  ETS  to  salvage  them. 

The  transcription  ot  the  student  response  data  into  machine-readable  form  was  achieved 
through  the  use  of  the  following  three  separate  s\  stems:  data  entry  (which  included  optical  mark 
recognition  (OMR)  and  image  scanning,  ICR.  and  key  entry),  data  validation  (edit),  and  data 
resolution. 


5.4.3  Data  Entry 

The  data  enlr\  process  was  the  fust  point  at  which  booklet-level  data  were  directly 
available  to  the  computer  s\stcm.  Depending  on  the  NAEP  document,  one  oi  three  methods  was 
used  to  transcribe  NAEP  data  to  a computerized  form.  The  data  on  seannable  documents  were 
collected  using  NCS  optical-scanning  equipment  that  also  captured  images  of  the  constructed- 
response  items  and  ICR  fields.  Nonscannable  materials  were  keyed  through  an  interactive  online 
system.  In  both  of  these  cases,  the  data  were  edited  and  suspect  cases  were  resolved  before 
further  processing. 

All  student  booklets,  questionnaires,  and  control  documents  were  seannable.  Throughout 
all  phases  of  processing,  the  student  booklets  were  hatched  by  grade  and  session  type.  The 
seannable  documents  were  then  transported  to  a slitting  area  where  the  folded  and  stapled  spine 
was  removed  from  the  document.  This  process  utilized  an  “intelligent  slitter”  to  prevent  slitting 
the  wrong  side  of  the  document.  The  documents  were  jogged  by  machine  so  that  the  registration 
edges  of  the  NAEP  documents  were  smoothly  aligned,  and  the  stacks  were  then  returned  to  the 
cart  to  be  scanned. 

Purine  the  scanning  process  (shown  in  Figure  each  seannable  NAEP  document  was 
umqueK  ulcnlitied  using  a pnnt-alter-sLan  number  consisting  ol  the  scan  batch  number,  the 
sequential  number  within  the  batch,  and  the  bar  code  IP  ol  the  booklet.  These  numbers  were 
printed  on  each  sheet  of  each  document  as  it  exited  the  scanner.  This  permitted  the  data  editors  to 


quickly  and  accurately  locate  specific  documents  during  the  editing  phase.  The  print-after-scan 
number  remained  with  the  data  record,  providing  a method  for  easy  identification  and  quick 
retrieval  of  any  document. 

The  data  values  were  captured  from  the  booklet  covers  and  Administration  Schedules 
and  were  coded  as  numeric  data.  Unmarked  fields  were  coded  as  blanks  and  editing  staff  were 
alerted  to  missing  or  uncoded  critical  data.  Fields  that  had  multiple  marks  were  coded  as  asterisks 
(*).  The  data  values  for  the  item  responses  and  scores  were  returned  as  numeric  codes.  'T  he 
multiple-choice  single  response  format  items  were  assigned  codes  depending  on  the  position  of 
the  response  alternative;  that  is*  the  first  choice  was  assigned  the  code  “I,"  the  sect  ad  ‘\2T  and 
* forth.  The  mark-all-ihat-apply  items  were  given  as  many  data  fields  as  response  alternatives; 
the  marked  choices  were  coded  as  while  the  unmarked  choices  were  recorded  as  blanks.  The 
images  of  constructed-response  items  were  saved  as  a digitized  computer  file.  The  area  of  the 
page  that  needed  to  be  clipped  was  defined  prior  to  scanning  through  the  document  definition 
process.  The  fields  from  unreadable  pages  were  coded  “X”  as  a flag  for  resolution  staff  to 
correct.  In  addition  to  capturing  the  student  responses,  the  barcode  identification  numbers  used 
to  maintain  process  control  were  decoded  and  transcribed  to  the  NAF.P  compute!* zed  data  file. 

As  the  scanning  program  completed  scanning  each  stack,  the  stack  was  removed  from  the 
output  hopper  and  placed  in  the  same  order  they  were  scanned  on  the  output  cart.  The  next  stack 
was  removed  from  the  input  cart  and  placed  into  the  input  hopper,  after  which  the  scanning 
resumed.  When  the  operator  had  completed  processing  the  last  slack  of  the  batch,  the  program 
was  terminated.  This  closed  the  dataset  that  automatically  became  available  for  the  data 
validation  (edit)  process.  The  scanned  documents  were  then  forwarded  to  a holding  area  in  case 
they  needed  to  be  retrieved  for  resolution  of  edit  errors. 

NCS  again  used  the  ICR  engine  to  read  various  hand  and  machine  printing  on  the  front 
cover  of  the  assessment  and  supervisor  documents  for  the  19%  NAHP  assessments.  Some 
information  from  scannable  student  documents,  such  as  the  Administration  Schedule,  the  Rosier 
of  Questionnaires,  and  some  questions  in  the  School  Characteristics  and  Policies  Questionnaires, 
were  read  by  the  ICR  engine  and  verified  by  an  online  key-entry  operator.  In  all,  the  ICR  engine 
read  approximately  15  million  characters.  The  ICR  engine  saved  NAIiP  field  staff  and  school 
personnel  a significant  amount  of  time  because  the\  no  longer  had  to  enter  this  data  by  gridding 
rows  and  columns  of  data. 

NCS  also  implemented  new  programs  that  allowed  the  scanners  to  read  imprinted  codes, 
known  as  2-out-of-5  codes,  that  were  printed  via  a Xerox  4280  printer  on  the  Administration 
Schedule.  These  2-out-of-5  codes  were  imprinted  at  the  same  time  the  booklet  ID  numbers  were 
printed  on  the  Administration  Schedule  and  identified  which  booklet  IDs  were  listed  on  that 
document  When  the  scanning  programs  were  unable  to  translate  the  2-out-of-5  codes  (thereby 
identifying  the  booklet  ID  numbers  on  the  document ) image  clips  of  the  booklet  ID  numbers 
were  displayed  to  online  editing  staff  for  verification.  This  eliminated  a significant  amount  of 
online  editing  time  needed  to  pioeess  iho  NAFP  assessments. 
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Figure  5-3 

1996  NAEP  State  Assessment 
Image  Scanning  Flow  Chart 
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To  provide  another  quality  check  on  the  image  scanning  and  scoring  system,  NCS  staff 
implemented  a quality  check  process  by  creating  a rubber  stamp  with  a valid  score  designated  on 
it.  In  this  way,  sample  responses  for  each  item  could  be  scanned  through  the  system  as  part  of  the 
regular  scanning  and  scoring  process.  An  example  of  the  stamp  used  is  given  below. 


Image  Scoring 
Quality  Assurance 
Sample 


Clerical  staff  stamped  blank  unused  booklets  with  mock  scores  and  sent  them  through 
the  scanning  process.  Each  unique  item  type  scored  via  the  image  system  had  two  quality  control 
stamps  per  valid  score. 

The  quality  control  booklets  were  batched  and  processed  together  with  student 
documents  of  the  same  type.  Because  all  of  a specific  item  were  batched  together  for 
transmission  to  the  scoring  facility,  the  quality  control-stamped  responses  were  integrated  with 
the  student  responses  and  transmitted  simultaneously  to  the  scoring  facility.  During  the  scoring 
process,  both  student  responses  and  the  quality  control  items  were  randomly  displayed  so  scores 
could  be  applied. 

When  a person  who  was  scoring  responses  (reader)  later  saw  the  quality  control  sample 
on  the  monitor  during  scoring,  he  or  she  was  to  notify  the  team  leader,  who  confirmed  the  score 
assigned  by  the  reader  was  the  score  listed  on  the  sample.  The  quality  control  booklets  were 
included  in  the  pool  of  all  items  to  be  drawn  from  for  the  25  percent  reliability  rescore. 

All  image  quality-assurance  documents  were  created  prior  to  the  beginning  of  scoring 
and  all  pre-determined  score  points  were  used.  Because  during  the  process  of  scoring,  valid  score 
points  can  be  changed  or  dropped  completely,  NCS  provided  ETS  with  documentation 
explaining  what  quality  control  documents  were  produced  and  which  score  points  on  these  items 
were  no  longer  valid.  When  an  image  quality  control  stamp  was  displayed  to  a reader  that 
contained  a score  point  that  was  no  longer  valid,  the  reader  gave  the  response  a score  point  of 
zero. 
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A key  entry  and  verification  process  was  used  to  make  corrections  to  the  teacher 
questionnaires  and  the  SD/LEP  student  questionnaires.  The  Falcon  system  that  was  used  to  enter 
these  data  is  an  online  data  entry  system  designed  to  replace  most  methods  of  data  input  such  as 
keypunch,  key-to-disk,  and  many  of  the  microcomputer  data  entry  systems.  The  terminal  screens 
were  uniquely  designed  for  NAEP  to  facilitate  operator  speed  and  convenience.  The  fields  to  be 
entered  were  titled  to  reflect  the  actual  source  document. 


5.4.4  Data  Validation 

Each  dataset  produced  by  the  scanning  system  contains  data  for  a particular  batch.  These 
data  had  to  be  validated  (or  edited)  for  type  and  range  of  response.  The  data-entry  and  resolution 
system  used  was  able  to  simultaneously  process  a variety  of  materials  from  all  age  groups. 
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subject  areas,  control  documents,  and  questionnaires  as  the  materials  were  submitted  to  the 
system  from  scannable  and  non-scannable  media. 

The  data  records  in  the  scan  file  were  organized  in  the  same  order  in  which  the  paper 
materials  were  processed  by  the  scanner.  A record  for  each  batch  header  preceded  all  data 
records  for  that  batch.  The  document  code  field  on  each  record  distinguished  the  header  record 
from  the  data  records. 

When  a batch-header  record  was  read,  a pre-edit  data  file  and  an  edit  log  were  generated. 
As  the  program  processed  each  record  within  a batch  from  the  scan  file,  it  wrote  the  edited  and 
reformatted  data  records  to  the  pre-edit  file  and  recorded  all  errors  on  the  edit  log.  The  data  fields 
on  an  edit  log  record  identified  each  data  problem  by  the  batch  sequence  number,  booklet  serial 
number,  section  or  block  code,  field  name  or  item  number,  and  data  value.  After  each  batch  had 
been  processed,  the  program  generated  a listing  or  online  edit  file  of  the  data  problems  and 
resolution  guidelines.  An  edit  log  listing  was  printed  at  the  termination  of  the  program  for  all 
non-image  documents.  Image  “clips”  requiring  editing  were  routed  to  online  editing  stations  for 
those  documents  that  were  image  scanned. 

As  the  program  processed  each  data  record,  it  first  read  the  booklet  number  and  checked 
it  against  the  session  code  for  appropriate  session  type.  Any  mismatch  was  recorded  on  the  error 
log  and  processing  continued.  The  booklet  number  was  then  compared  against  the  first  three 
digits  of  the  student  identification  number.  If  they  did  not  match,  a message  was  written  on  the 
error  log.  The  remaining  booklet  cover  fields  were  read  and  validated  for  the  correct  range  of 
values.  The  school  codes  had  to  be  identical  to  those  on  the  Process  Control  System  record.  All 
data  values  that  were  out  of  range  were  read  “as  is”  but  were  flagged  as  suspect.  All  data  fields 
that  were  read  as  asterisks  (*)  were  recorded  on  the  edit  log  or  online  edit  file. 

Document  definition  files  described  each  document  as  a series  of  blocks  that  in  turn  were 
described  as  a series  of  items.  The  blocks  in  a document  were  transcribed  in  the  order  that  they 
appeared  in  the  document.  Each  block's  fields  were  validated  during  this  process.  If  a document 
contained  suspect  fields,  the  cover  information  was  recorded  on  the  edit  log  along  with  a 
description  of  the  suspect  data.  The  edited  booklet  cover  was  transferred  to  an  output  buffer  area 
within  the  program.  As  the  program  processed  each  block  of  data  from  the  dataset  record,  it 
appended  the  edited  data  fields  to  the  data  already  in  this  buffer. 

The  program  then  cycled  through  the  data  area  corresponding  to  the  item  blocks.  The 
task  of  translating,  validating,  and  reporting  errors  for  each  data  field  in  each  block  was 
performed  by  a routine  that  required  only  the  block  identification  code  and  the  string  of  input 
data.  This  routine  had  access  to  a block  definition  file  that  had,  for  each  block,  the  number  of 
fields  to  be  processed,  and,  for  each  field,  the  field  type  (alphabetic  or  numeric),  the  field  width 
in  the  data  record,  and  the  valid  range  of  values.  The  routine  then  processed  each  field  in 
sequence  order,  performing  the  necessary  translation,  validation,  and  reporting  tasks. 

The  first  of  these  tasks  checked  for  the  presence  of  blanks  or  asterisks  (*)  in  a critical 
field.  These  were  recorded  on  the  edit  log  or  online  edit  file  and  processing  continued  with  the 
next  field.  No  action  was  taken  on  blank  fields  for  multiple-choice  items  because  the  asterisk 
code  indicated  a non-response.  The  field  was  validated  for  range  of  response,  and  any  values 
outside  of  the  specified  range  were  recorded  on  the  edit  log  or  online  edit  file.  The  program  used 
the  item-type  code  to  make  a further  distinction  among  constructed-rcsponse  item  scores  and 
other  numeric  data  fields. 
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Moving  the  translated  and  edited  data  field  into  the  output  buffer  was  the  last  task 
performed  in  this  phase  of  processing.  When  the  entire  document  was  processed,  the  completed 
string  of  data  was  written  to  the  data  file.  When  the  program  encountered  the  end  of  a file,  it 
closed  the  dataset  and  generated  an  edit  listing  for  non-image  and  key-entered  documents.  Image- 
scanned  items  that  required  correction  were  displayed  at  an  online  editing  terminal. 


5.4.5  Editing  for  Non-Image  and  Key-Entered  Documents 

Throughout  the  system,  quality  procedures  and  software  ensured  that  the  NAEP  data 
were  correct.  All  student  documents  on  the  Administration  Schedule  were  accounted  for,  as 
receipt  control  personnel  checked  that  the  materials  were  undamaged  and  assembled  correctly. 
The  machine  edits  performed  during  data  capture  verified  that  each  sheet  of  each  document  was 
present  and  that  each  field  had  an  appropriate  value.  All  batches  entered  into  the  system,  whether 
key-entered  or  machine-scanned,  were  edited  for  errors. 

Data  editing  took  place  after  these  checks.  This  consisted  of  a computerized  edit  review 
of  each  respondent’s  document  and  the  clerical  edits  necessary  to  make  corrections  based  upon 
the  computer  edit.  This  data-editing  step  was  repeated  until  all  data  were  correct. 

The  first  phase  of  data  editing  was  designed  to  validate  the  population  and  ensure  that  all 
documents  were  present.  A computerized  edit  list,  produced  after  NAEP  documents  were 
scanned  or  key  entered,  and  all  the  supporting  documentation  sent  from  the  field  were  used  to 
perform  the  edit  function.  The  hard-copy  edit  list  contained  all  the  vital  statistics  about  the  batch: 
number  of  students,  school  code,  type  of  document,  assessment  code,  suspect  cases,  and  record 
serial  numbers.  Using  these  inputs,  the  data  editor  verified  that  the  batch  had  been  assembled 
correctly  and  that  each  school  number  was  correct. 

During  data  entry,  counts  of  processed  documents  were  generated  by  type.  These  counts 
were  compared  against  the  information  captured  from  the  Administration  Schedules.  The  number 
of  assessed  and  absent  students  processed  had  to  match  the  numbers  indicated  on  the  Process 
Control  System. 

In  the  second  phase  of  data  editing,  experienced  editing  staff  used  a predetermined  set  of 
specifications  to  review  the  field  errors  and  record  necessary  corrections  to  the  student  data  file. 
The  same  computerized  edit  list  used  in  phase  one  was  used  to  perform  this  function.  The  editing 
staff  reviewed  the  computer-generated  edit  log  and  the  area  of  the  source  document  that  was 
noted  as  being  suspect  or  as  containing  possible  errors.  The  composition  of  the  field  was  shown 
in  the  edit  box.  The  editing  staff  checked  this  piece  of  information  against  the  NAEP  source 
document.  At  that  point,  one  of  the  following  took  place: 

Correctable  error.  If  the  error  was  correctable  by  the  editing  staff  as  per  the 
editing  specifications,  the  correction  was  noted  on  the  edit  log  for  later 
correction  via  key-entry. 

Alert.  If  an  error  was  not  correctable  as  per  the  specifications,  an  alert  was  issued 
to  NAEP  project  staff  for  resolution.  Once  the  correct  information  was  obtained, 
the  correction  was  noted  on  the  edit  log  for  key-entry  correction. 
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N on-correctable  error . If  a suspected  error  was  found  to  be  correct  as  stated  and 
no  alteration  was  possible  according  to  the  source  document  and  specifications, 
the  programs  were  tailored  to  allow  this  information  to  be  accepted  into  the  data 
record.  No  corrective  action  was  taken. 

The  corrected  edit  log  was  then  forwarded  to  the  key-entry  staff  for  processing.  When  all 
corrections  were  entered  and  verified  for  a batch,  an  extract  program  pulled  the  corrected  records 
into  a mainframe  dataset.  At  this  point,  the  mainframe  edit  program  was  initiated.  The  edit 
criteria  were  again  applied  to  all  records.  If  there  were  further  errors,  a new  edit  listing  was 
printed  and  the  cycle  was  repeated. 

When  the  edit  process  produced  an  error-free  file,  the  booklet  ID  number  was  posted  to 
the  NAEP  tracking  file  by  age,  assessment,  and  school.  This  permitted  NCS  staff  to  monitor  the 
NAEP  processing  effort  by  accurately  measuring  the  number  of  documents  processed  by  form. 
The  posting  of  booklet  IDs  also  ensured  that  a booklet  ID  was  not  processed  more  than  once. 


5.4,6  Data  Validation  and  Editing  of  Image-Processed  Documents 

The  paper  edit  log  for  key-entered  documents  was  replaced  by  online  viewing  of  suspect 
data  for  all  image-processed  documents.  For  rapid  resolution,  the  edit  criteria  for  each  item  in 
question  appeared  on  the  screen  along  with  the  suspect  item.  Corrections  were  made 
immediately.  The  system  employed  an  edit/verify  system  that  ultimately  meant  that  two  different 
people  viewed  the  same  suspect  data  and  operated  on  it  separately.  The  “verifier”  made  sure  the 
two  responses  (one  from  either  the  entry  operator  or  the  ICR  engine)  were  the  same  before  the 
system  accepted  that  item  as  being  correct.  The  verifiers  could  either  overrule  or  agree  with  the 
original  correction  made  if  the  two  did  not  match.  If  the  editor  could  not  determine  the 
appropriate  response,  he  or  she  escalated  the  suspect  situation  to  a supervisor.  For  errors  or 
suspect  information  that  could  not  be  resolved  by  supervisory  staff,  a product- line  queue  was 
created.  This  allowed  supervisors  to  escalate  edits  to  project  staff  for  resolution.  By  having  this 
product-line  queue,  project  staff  were  able  to  quickly  locate  edit  clips  within  the  image  system, 
speeding  up  the  resolution  process. 

Once  an  entire  batch  was  through  the  edit  phase,  it  became  eligible  for  the  count- 
verification  phase.  The  Administration  Schedule  data  were  examined  systematically  for  booklet 
IDs  that  should  have  been  processed  (assessed  administration  codes).  All  documents  under  that 
Administration  Schedule  were  then  inspected  to  ensure  that  all  of  the  booklets  were  included. 

With  the  satisfactory  conclusion  of  the  count- verification  phase,  the  edited  batch  file  was 
uploaded  to  the  mainframe,  where  it  went  through  yet  another  edit  process.  A paper  edit  log  was 
produced  and,  if  errors  remained,  was  forwarded  to  another  editor.  When  this  paper  edit  was 
satisfied,  the  Process  Control  System  and  Workflow  Management  System  were  updated.  Because 
there  was  a possible  time  lag  between  a clean  edit  in  the  image  system  and  a clean  edit  in  the 
mainframe  systems,  the  batch  was  not  archived  until  48  hours  after  the  image  edit  phase  was 
completed. 


5.4.7  Data  Transmission 


Due  to  the  rapid  pace  of  scoring  on  an  item-by-item  basis,  the  NCS  scoring  specialists 
found  it  necessary  to  continually  monitor  the  status  of  work  available  to  the  readers  and  plan  the 
scoring  schedule  several  weeks  in  advance.  On  Wednesday  of  each  week,  the  NCS  performance 
assessment  specialist  in  charge  of  each  subject  area  planned  the  next  two  weeks’  schedule.  That 
information  was  then  provided  to  the  person  in  charge  of  downloading  data  to  the  scoring  center. 
By  planning  the  scoring  schedule  two  weeks  in  advance,  the  scoring  specialists  were  able  to 
ensure  that  readers  would  have  sufficient  work  for  at  least  one  week,  after  which  the  next 
download  would  occur  to  supplement  the  volume  of  any  unscored  items  and  add  an  additional 
week’s  work  to  the  pool  of  items  to  score.  Additionally,  by  scheduling  two  weeks’  data 
transmission,  flexibility  was  added  to  the  scoring  schedule,  making  it  possible  to  implement  last- 
minute  changes  in  the  schedule  once  the  items  had  been  delivered  to  the  scoring  center. 
Depending  on  the  number  of  items  to  be  transmitted,  the  actual  downloading  was  conducted  on 
Friday  or  was  divided  into  two  smaller  sessions  for  Thursday  and  Friday  download.  By  the  first 
week  of  May  1996,  there  was  sufficient  space  on  the  scoring  servers  to  load  all  remaining 
unscored  items  to  the  scoring  center. 

Delivery  of  data  to  the  scoring  center  was  accomplished  via  several  T1  transmission  lines 
linking  the  mainframe  computers  and  the  NAEP  servers  at  the  document-scanning  site  in  the 
NCS  main  facility  with  the  scoring  servers  dedicated  to  distributing  work  to  the  professional 
readers  at  the  scoring  center.  The  actual  task  of  scheduling  items  for  downloading  was 
accomplished  using  a code  written  by  the  Image  Software  Development  team.  This  code  enabled 
the  person  scheduling  the  download  to  choose  a team  of  readers  and  select  the  scheduled  items 
from  a list  of  all  items  that  that  team  would  be  scoring  throughout  the  scoring  project.  This 
process  was  repeated  for  all  teams  of  readers  until  all  anticipated  work  was  scheduled.  Once  this 
task  was  completed,  the  scheduled  job  was  tested  to  determine  if  there  was  sufficient  free  disk 
space  on  the  servers  at  the  scoring  center.  If  for  any  reason  sufficient  disk  space  was  not 
available,  scheduled  items  could  be  deleted  from  the  batch  individually  or  as  a group  until  the 
scheduled  batch  job  could  accommodate  all  items  on  the  available  disk  space  at  the  scoring 
center.  Once  it  was  determined  that  sufficient  disk  space  was  available,  transmission  of  student 
responses  commenced.  Data  transmission  was  typically  accomplished  during  off-shift  hours  to 
minimize  the  impact  on  system-load  capacity. 


5.5  PROFESSIONAL  SCORING 
5.5.1  Overview 

Scoring  of  the  1996  NAEP  State  Assessment  constructed-response  items  was  conducted 
using  NCS’s  imaging  technology.  All  1996  responses  were  scored  online  by  readers  working  at 
image  stations.  The  logistical  problems  associated  with  handling  large  quantities  of  student 
booklets  were  removed  for  those  items  scored  on  the  image  system. 
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One  of  the  greatest  advantages  image  technology  presented  for  NAEP  scoring  v/as  in  the 
area  of  soiling  and  distributing  work  to  scorers.  All  student  responses  for  a particular  item, 
regardless  of  where  spiraling  had  placed  that  item  in  the  various  booklet  forms,  were  grouped 
together  tor  presentation  to  a team  of  readers.  This  allowed  training  to  be  conducted  one  item  at  a 
time,  rather  than  in  blocks  of  related  items,  thus  focusing  readers'  attention  on  the  complexities 
of  a single  item. 

A number  ot  tools  built  into  the  system  allowed  table  leaders  and  trainers  to  closely  and 
continuously  monitor  reader  performance.  A detailed  discussion  of  these  tools  can  be  found  later 
in  this  chapter. 

The  system  automatically  routed  six  percent  of  student  responses  to  other  members  of 
the  team  for  second  scoring.  Readers  were  given  no  indication  of  whether  the  response  had  been 
scored  by  another  reader,  thereby  making  the  second  scoring  truly  blind.  On-demand,  real-time 
reports  on  interreader  reliability  (drawn  from  those  items  that  were  second-scored)  presented 
extremely  valuable  information  on  team  and  individual  scoring.  Information  on  adjacent  and 
perfect  agreement,  score  distribution,  and  quantity  of  responses  scored  were  continuously 
available  for  consultation.  Similarly,  back-reading  of  student  responses  could  be  accomplished  in 
an  efficient  and  timely  manner.  Also,  table  leaders  were  able  to  read  a large  percentage  of 
responses,  evaluating  the  appropriateness  and  accuracy  of  the  scores  assigned  by  readers  on  their 
teams. 


Project  management  tools  assisted  table  leaders  in  making  well-informed  decisions.  For 
example,  knowledge  of  the  precise  number  of  responses  remaining  to  be  scored  for  a particular 
item  allowed  table  leaders  to  determine  the  least  disruptive  times  for  lunch  breaks. 

Concerns  about  possible  reader  fatigue  or  other  problems  that  might  result  from  working 
continuously  at  a computer  terminal  proved  unfounded.  Both  readers  and  table  leaders  responded 
vvith  enthusiasm  to  the  system,  remarking  on  the  ease  with  which  student  responses  could  be  read 
and  on  the  increased  sense  of  professionalism  they  felt  in  working  in  this  technological 
environment.  Readers  took  periodic  breaks,  in  addition  to  their  lunch  break,  to  reduce  the  degree 
of  visual  fatigue.  Readers  were  grouped  in  teams  of  9 to  14  readers  per  team;  each  team  working 
with  a specific  table  leader. 


5.5.2  Training  Paper  Selection 

A pool  of  papers  to  be  used  during  training  for  the  national  main  assessment  was  selected 
by  NCS  staff  in  March  19%.  During  the  interview  process,  NCS  performance  assessment 
specialists  identified  those  candidates  with  team  leader  potential.  Individuals  recruited  to  be  team 
leaders  during  the  actual  scoring  were  asked  to  select  student  responses  to  send  to  HTS 
assessment  division  subject  specialists,  who  created  the  master  training  set.  Team  leaders  were 
used  for  this  task  because  it  gave  them  the  advantages  of  working  on  specific  items,  learning  the 
make-up  of  the  various  booklets,  learning  the  terminology,  and  understanding  the  processing  of 
the  booklets  at  NCS.  This  was  especially  important  in  19%,  because  most  scoring  activities 
occurred  via  the  image  processing  system. 


Generally,  the  training  set  for  each  short  (two-  or  three-point)  item  included  40  papers: 

• 10  anchor  papers 

• 20  practice  papers 

• 5 papers  in  Calibration  Set  #1 

• 5 papers  in  Calibration  Set  #2 

Generally,  the  training  set  for  each  extended  (four-  or  five-point)  item  included  85 

papers: 


• 15  anchor  papers 

• 40  practice  papers 

• 10  papers  in  each  of  two  qualification  sets 

• 5 papers  in  Calibration  Set  #1 

• 5 papers  in  Calibration  Set  #2 

Anchor  papers,  or  sets,  are  those  papers  that  represent  the  best  examples  of  each  score 
point.  They  are  used  to  illustrate  the  scoring  guide  so  that  the  reader  can  return  to  this  set  and 
compare  it  with  student  responses  during  scoring.  Practice  papers,  or  sets,  include  the  remainder 
of  the  scored  examples,  excluding  the  scores,  so  that  the  reader  can  practice  on  some  student 
responses  prior  to  scoring.  The  purpose  is  to  elicit  discussion  and  give  scorers  a chance  to  ask 
questions.  Qualification  sets  are  used  by  the  trainer  to  ensure  that  each  reader  has  understood  the 
scoring  guide  and  can  apply  it  to  student  documents.  Similar  to  practice  papers,  the  scores  are 
masked  so  the  reader  can  assign  a score.  A predetermined  number  of  scores  must  be  correct  for 
the  reader  to  remain  on  the  scoring  project.  Calibration  sets  are  used  after  a long  break  in  scoring 
has  occurred  (e.g.,  after  lunch  in  the  early  days  of  a project,  or  first  thing  in  the  morning)  to 
ensure  that  the  readers  review  the  scoring  guide  and  the  anchor  papers,  and  to  prevent  the  scorers 
from  drifting  to  the  middle  range  of  possible  score  points. 

To  ensure  that  the  HTS  assessment  specialist  would  have  a wide  range  of  student 
responses  to  encompass  all  score  points,  NCS  personnel  copied  approximately  125  papers  foi 
each  five-point  item,  100  papers  for  each  four-point  item,  75  papers  for  each  three-point  item, 
and  50  papers  for  each  two-point  item.  To  ensure  that  training  papers  represented  the  range  of 
responses  obtained  from  the  sample  population,  NCS  personnel  selected  papers  randomly  from 
across  the  sample.  The  student  identifier  (barcode)  was  written  on  the  copy  and  NCS  team 
leaders  assigned  tentative  scores  to  the  responses.  The  responses  were  numbered  sequentially, 
copied,  and  sent  via  overnight  deliver)  to  HTS.  When  the  training  packet  was  compiled,  the  hTS 
assessment  specialist  faxed  the  composition  of  the  packets  back  using  the  sequential  numbeis. 
HTS  staff  kept  its  copy  of  the  training  sets. 

From  the  faxed  sheets,  packets  were  created  for  each  item  using  the  original  copies  of  the 
student  responses.  These  packets  were  then  forwarded  to  the  NC  S communication  center  for 
copying,  and  stored  for  the  team's  use  in  training.  HTS  also  sent  the  most  up-to-date  version  ol 
the  training  packet  to  the  NCS  scoring  center  for  each  item  to  be  included  in  the  scoring  guide. 
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5.5.3  General  Training  Guidelines 


ETS  and  NCS  personnel  conducted  training  for  the  constructed-response  items  on  an 
item-by-item  basis,  so  that  each  item  could  be  scored  immediately  after  training. 

In  all,  14  table  leaders  and  196  readers  worked  from  March  13  to  May  5,  1996,  to 
complete  scoring  for  the  1996  NAEP  State  Assessments.  Each  member  of  a team  received  a copy 
of  the  stimulus  and  training  materials  for  the  items  that  his  or  her  team  would  be  scoring.  Before 
training,  each  team  member  became  famibar  with  those  materials  under  the  guidance  of  the 
trainer  who  explained  the  anchor  papers,  exemplifying  the  various  score-point  levels.  Next,  ETS 
and  NCS  staff  (the  trainers)  conducted  training  sessions  to  explain  the  anchor  papers, 
exemplifying  the  various  score  point  levels.  The  team  proceeded  with  each  member  scoring  the 
practice  papers,  and  then  discussing  those  papers  as  a group  while  the  trainer  clarified  issues  and 
answered  questions.  The  papers  selected  for  each  training  set  were  chosen  to  illustrate  a range 
from  easily  classifiable  responses  to  borderline  responses  for  each  score  point. 

When  the  trainer  was  confident  the  readers  were  ready  to  begin  scoring  short  constructed 
responses,  the  table  leader  signaled  the  system  to  release  the  responses  to  the  team  members  who 
had  successfully  completed  training.  For  extended  constructed-response  items,  each  team 
member  was  given  a qualifying  set  that  had  been  prescored  by  the  trainer  in  conjunction  with  the 
table  leader.  Readers  were  required  to  score  an  exact  match  on  80  percent  of  the  items  in  order  to 
qualify  for  scoring.  If  a reader  failed  on  the  first  attempt,  the  trainer  discussed  the  discrepant 
scores  with  the  reader  and  administered  a second  qualifying  set.  Again,  80  percent  exact 
agreement  was  required  to  score  the  item.  During  the  beginning  stages  of  scoring,  the  team 
members  discussed  student  responses  with  the  trainer  and  table  leader  to  ensure  that  issues  not 
addressed  in  training  were  handled  in  the  same  manner  by  all  team  members. 

After  the  initial  training,  readers  scored  the  items,  addressing  questions  to  the  table 
leader  and/or  trainer  when  appropriate.  Depending  upon  the  number  of  responses,  length  of 
responses  and  complexity  of  the  rubric,  scoring  of  an  individual  item  ranged  anywhere  from  one- 
half  hour  to  two  weeks.  Whenever  a break  longer  than  15  minutes  occurred  in  scoring,  each  team 
member  received  a set  of  calibration  papers  that  had  been  prescored  by  the  trainer  and  table 
leader.  Each  team  member  scored  the  calibration  set  individually,  and  then  the  team  discussed 
the  papers  to  ensure  against  scorer  drift. 


5.5.4  Table  Leader  Utilities  and  Reliability  Reports 

Among  the  many  advantages  of  the  image  scoring  system  is  the  ease  with  which 
workflow  to  readers  can  be  regulated  and  scoring  can  be  monitored.  One  of  the  utilities  at  a table 
leader's  disposal  was  a qualification  algorithm  used  after  training  on  extended  constructed- 
response  items.  This  algorithm  was  used  to  compare  readers'  scores  with  the  scores  agreed  upon 
by  the  trainer  and  the  table  leader  for  the  qualifying  set  of  responses.  The  table  leader  would  give 
identical  paper-copy  qualification  packets  to  each  reader.  These  packets  contained  ten  student 
responses  to  be  independently  scored  by  the  readers.  After  the  readers  finished,  the  table  leader 
would  enter  each  reader's  scores  into  the  computer  for  tabulation.  The  computer  would  calculate 
each  reader's  percent  of  exact,  adjacent,  and  non-adjacent  agreement  with  the  master  key.  If  a 
reader  attained  a percent  of  exact  agreement  above  a pro-determined  threshold  (usually  80r/r),  the 
reader  would  be  allowed  to  score.  Readers  not  attaining  the  pro-determined  threshold  were 
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handled  on  a case-by-case  basis— typically  receiving  individual  training  by  the  trainer  or  the 
NCS  table  leader  before  being  allowed  to  score.  A table  leader  could  also  cancel  a reader’s 
qualification  to  score  an  item  if  review  of  a reader’s  work  indicated  inaccurate  scoring  and  that 
supplemental  training  was  necessary  after  scoring  had  begun. 

After  scoring  began,  NCS  table  leaders  reviewed  each  reader’s  progress  using  a 
backreading  utility  that  allowed  the  table  leader  to  review  papers  scored  by  each  reader  on  the 
team.  Typically,  a table  leader  reviewed  responses  scored  by  each  reader  at  the  same  rate  at 
which  second  scoring  occurred  (i.e.,  six  percent  for  items  with  both  state  and  national  samples 
and  more  for  items  with  only  a national  sample).  Table  leaders  made  certain  to  note  the  score  the 
reader  awarded  each  response  as  well  as  the  score  a second  reader  gave  that  same  paper.  This 
was  done  as  an  interreader  reliability  check.  Alternatively,  a table  leader  could  choose  to  review 
all  1 .ponses  given  a particular  score  to  determine  if  the  team  as  a whole  was  scoring 
consistently.  Both  of  these  review  methods  used  the  same  display  screen  and  showed  the  ID 
number  of  the  reader  and  the  scores  awarded.  If  the  table  leader  disagreed  with  the  score  given  an 
item,  he  or  she  discussed  it  with  the  reader  for  possible  correction.  Replacement  of  scores  by  the 
table  leader  was  done  only  with  the  knowledge  and  approval  of  the  reader,  thereby  serving  as  a 
learning  experience  for  the  reader.  In  the  case  where  the  response  was  second  scored,  neither 
score  was  changed. 

The  table  leaders  were  able  to  monitor  workflow  using  a status  tool  that  displayed  the 
number  of  items  scored,  the  number  of  items  first-scored  that  still  needed  to  be  second-scored, 
the  number  of  items  remaining  to  be  second-scored,  and  the  total  number  of  items  remaining  to 
be  scored.  This  allowed  the  team  leaders  and  performance  assessment  specialists  to  accurately 
monitor  the  rate  of  scoring  and  to  estimate  the  time  needed  for  completion  of  the  various  phases 
of  scoring. 

The  reliability  information  about  the  constructed-response  items  used  in  the  NAEP  scale 
will  be  discussed  in  The  NAEP  /996  Technical  Report  (Allen.  Carlson,  & Xelenak,  1998). 


5.5.5  Main  and  State  Mathematics  Assessments 

It  is  important  to  note  that  the  student  responses  in  the  fourth-  and  eighth-grade 
mathematics  assessments  were  scored  concurrently  for  the  national  and  the  state  samples. 
Another  advantage  of  image-based  item-by-item  scoring  is  that  the  comparability  of  the  scoring 
of  the  two  samples  is  ensured  because  all  responses  are  scored  simultaneously  and  in  a manner 
that  makes  it  impossible  to  know  which  sample  any  individual  response  belongs  to.  Because  of 
this,  the  following  discussion  addresses  both  national  (main)  and  state  mathematics  assessments 
Training  procedures  for  the  scoring  of  mathematics  items  followed  the  steps  outlined  in  Section 
5.5.3. 


5.5.6  Scoring  the  Main  and  State  Mathematics  Assessments 

Each  constructed-response  item  had  a unique  scoring  standard  that  identified  the  range  ol 
possible  scores  for  the  item  and  defined  the  criteria  to  be  used  in  evaluating  the  students' 
responses.  Point  values  were  assigned  with  the  following  general  meanings; 
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Dichotomous  items  from  the  1992  assessment2 


• 1 = Unacceptable 

• 7 = Acceptable 

Three-point  items  developed  during  the  1993  field  test 

• 1 = Evidence  of  little  or  no  understanding 

• 2 = Evidence  of  partial  understanding 

• 3 = Evidence  of  full  understanding 

Four-  and  five-point  items 

• 1 = No  evidence  of  understanding 

• 2 = Evidence  of  minimal  understanding 

• 3 = Evidence  of  partial  understanding 

• 4/5  = Evidence  of  satisfactory/extended  understanding 

The  scores  for  these  items  also  included  a 0 for  no  response,  8 for  an  erased  or  crossed-out 
response,  and  a 9 for  any  response  found  to  be  unratable  (i.e.,  illegible,  off-task,  responses 
written  in  a language  other  than  English,  responses  of  “I  don’t  know,”  or  refusal  to  participate). 

During  scoring,  the  table  leaders  compiled  notes  on  various  responses  for  the  readers’ 
reference  and  guidance  and  for  the  permanent  record.  In  addition,  trainers  were  accessible  for 
consultation  in  interpreting  the  guides  for  unusual  or  unanticipated  responses.  Each  item  was 
scored  by  a single  team  immediately  after  training  for  that  item.  The  table  leaders  conducted 
constant  online  back-reading  of  all  team  members’  work  throughout  the  scoring  process,  bringing 
to  the  attention  of  each  reader  any  problems  relating  to  scoring.  When  deemed  appropriate, 
scoring  issues  were  discussed  among  the  team  as  a whole.  Table  leaders  also  monitored  the 
number  of  responses  scored  and  individual  and  team  reliability  figures  throughout  the  course  of 
scoring. 


Grades  4 and  8 items  came  from  both  national  and  state-by-state  samples.  Responses 
were  delivered  by  image  in  such  a way  that  the  student  demographics  were  unknown  to  the 
reader.  Thus,  readers  did  not  know  from  which  sample  any  given  item  came  when  it  appeared  on 
the  screen.  In  the  case  of  overlap  items,  all  readers  scored  responses  at  both  grade  levels. 


2 Some  dichotomous  items  have  other  unacceptable  responses  tracked  as  a 2.  3,  4,  or  5,  while  some  have  other  correct 
responses  tracked  as  cither  a 6 or  7. 
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5.6  DATA  DELIVERY 


The  1996  NAEP  assessment  data  collection  resulted  in  several  classes  of  data 
files — student  background,  school,  teacher,  weights,  SD/LEP  student,  student/teacher  match,  and 
student-response  information.  Student-response  information  included  response  data  from  all 
assessed  students  in  1996.  Data  resolution  activities  occurred  prior  to  the  submission  of  data  files 
to  ETS  and  Westat  to  resolve  any  irregularities  that  existed.  This  section  details  additional  steps 
performed  before  creating  the  final  data  files  to  ensure  capture  of  the  most  complete  and  accurate 
information. 

An  important  quality-control  component  of  the  image-scoring  system  was  the  inclusion, 
with  a student's  response  to  one  item,  of  an  exact  copy  of  the  student  edit  record,  including  the 
student  booklet  ID  number,  with  every  image  of  a student’s  response  to  a constructed-response 
item.  This  information  was  used  to  identify  the  file  within  the  image-scoring  system.  These  edit 
files  also  remained  in  the  main  data  files  residing  on  the  NCS  mainframe  computer.  By  attaching 
this  information  to  a student’s  response,  exact  matching  of  scores  assigned  to  constructed- 
response  items  and  all  other  data  for  each  individual  student  was  guaranteed,  because  the  booklet 
ID  for  each  image  was  part  of  every  image  file.  This  ensured  scores  were  applied  to  the  correct 
student’s  record  on  the  mainframe. 

When  all  the  responses  for  an  individual  item  had  been  scored,  the  system  automatically 
submitted  all  item  scores  assigned  during  scoring,  along  with  their  student  edit  records,  to  a 
queue  to  be  transmitted  to  the  mainframe.  Project  staff  then  initiated  a system  job  to  transmit  all 
scoring  data  to  be  matched  with  the  original  student  records  on  the  mainframe.  A custom  edit 
program  matched  the  edit  records  of  the  scoring  files  to  those  of  the  original  edit  records  on  the 
mainframe.  As  matches  were  confirmed,  the  scores  were  applied  to  those  individual  files.  After 
completion  of  this  stage,  all  data  collected  for  an  individual  student  was  located  in  one  single  and 
complete  record/file  identified  by  the  student  edit  record. 

NCS  processed  the  SD/LEP  Student  Questionnaires  via  OMR  scanning.  Edits  performed 
on  the  questionnaires  assured  that  responses  to  questions  fell  within  the  valid  range  for  that 
question.  SD/LEP  q estionnaires  were  then  matched  to  a student  record.  SD/LEP  questionnaires 
that  were  not  matched  to  a student  document  were  cross-referenced  with  the  corresponding 
Administration  Schedule,  Roster  of  Questionnaire,  and  student  data  files  to  correct,  if  necessary . 
the  information  needed  to  result  in  a match. 

In  1996,  NCS  continued  to  use  ICR  technology  to  capture  percentage  figures  written  by 
school  personnel  directly  in  boxes  on  the  School  Characteristics  and  Policies  Questionnaires 
rather  than  requiring  the  school  official  to  grid  ovals  in  a matrix.  The  data  were  then  verified  by 
an  edit  operator. 

The  same  processes  that  were  followed  in  previous  cycles  were  used  in  1996  to  achieve 
the  best  possible  student/teacher  match  rate.  The  first  step  was  to  identity  Teacher 
Questionnaires  not  returned  to  NCS  for  processing  so  as  to  exclude  from  the  matching  process 
the  students  of  these  teachers.  Student  identification  numbers  that  were  not  matched  to  a Teacher 
Questionnaire  were  cross-referenced  with  the  corresponding  Administration  Schedule  and  Roster 
of  Teacher  Questionnaires  to  verify  (and  change,  if  necessary)  the  teacher  number,  teacher 
period,  and  questionnaire  number  recorded  on  these  control  documents.  The  NAhP  school 
numbers  listed  on  the  Roster  ol  Questionnaires  and  Teacher  Questionnaire  were  verified  and 


corrected,  if  necessary.  Once  these  changes  were  made,  any  duplicate  teacher  numbers  existing 
within  a school  were,  if  possible,  cross-referenced  for  resolution  with  the  Rosters  of 
Questionnaires.  Because  this  information  was  located  together  on  a single,  central  control 
document,  the  ability  to  match  and  resolve  discrepant  or  missing  fields  was  simplified. 

After  all  data  processing  activities  were  completed,  data  cartridges  and/or  diskettes  were 
created  and  shipped  via  overnight  delivery  to  ETS  and/or  Westat,  as  appropriate.  NCS  maintains 
a duplicate  archive  file  for  security/backup  purposes. 


5.7  MISCELLANEOUS 

5.7.1  Storage  of  Documents 

After  the  batches  of  image-scanned  documents  had  successfully  passed  the  editing 
process,  they  were  sent  to  the  warehouse  for  storage.  Due  to  the  large  number  of  rescore  projects 
done  with  NAEP  material,  the  documents  were  unspiraled  and  sequenced  by  grade  and  booklet 
type  after  all  of  the  processing/scoring  was  completed.  Unspiraled  and  sequenced  booklets  were 
then  assigned  a new  inventory  number  by  grade  and  booklet  type  and  sent  back  to  the  warehouse 
for  storage  The  storage  locations  of  all  documents  were  recorded  on  the  inventory  control 
system.  Unused  materials  were  sent  to  temporary  storage  to  await  completion  of  the  entire 
assessment.  Once  the  assessment  was  complete,  NCS  received  authorization  from  ETS  to  salvage 
unused  materials  after  determining  that  a sufficient  quantity  of  each  form  type  was  retained 
permanently. 


5.7.2  Quality  Control  Documents 

ETS  requires  that  a random  sample  of  booklets  and  the  corresponding  scores/scoring 
sheets  be  pulled  for  an  additional  quality-control  check  that  verifies  the  accuracy  and 
completeness  of  the  data.  For  image-scanned  documents,  a scoring  sheet  is  not  used,  so  ETS  uses 
scores  sent  to  them  on  a data  tape  to  verify  the  accuracy  of  applied  score  During  the  scoring  of 
mathematics,  a selected  number  of  image-processed  booklets  were  paper  scored.  If  any  of  the 
random  sample  of  mathematics  booklets  used  for  paper  scoring  were  selected  as  quality  control 
documents,  the  scoring  sheet  was  also  sent  to  ETS.  All  of  these  documents  were  selected  prior  to 
sending  the  booklets  to  storage.  A random  sample  of  all  the  questionnaires  used  in  the  1996 
NAEP  assessment  was  also  sent  to  ETS. 


5.7.3  Alert  Analysis 

Table  5-5  identifies  the  different  types  of  alerts  to  problems  that  were  encountered  in  the 
processing  of  NAEP  data.  For  the  1996  St. tie  Assessment,  there  was  a total  of  3,812  alerts. 

Discrepancies  were  found  in  the  receiving  process  that  did  not  require  an  alert  to  be 
issued  to  Westat.  They  did  require  a great  deal  of  effort  by  the  opening  staff  to  resolve  in  order  to 
provide  the  most  complete  and  accurate  information.  These  are  referred  to  as  “info  alerts.”  These 
were  categorized  and  codes  were  assigned  to  them.  They  are  listed  in  the  left-hand  column  of 
Table  5-5. 
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Even  though  receipt-control  staff  were  well  trained  in  the  resolution  of  many  situations, 
there  were  some  problems  that  required  resolution  by  NCS  NAEP  product  line  staff.  These  are 
referred  to  as  “problem  alerts  ” The  various  types  of  problem  alerts  were  also  categorized  and 
coded.  They  are  listed  in  the  right-hand  column  of  Table  5-5.  For  any  unusual  situations,  Westat 
was  contacted  to  help  with  the  resolution  of  the  alert. 

Table  5-5 

Alerts  for  the  1996  National  and  State  Assessments 


Information  Alerts 

Code  5°.  not  written  on  Administration 
Schedules 

The  yes/no  box  not  gridded  on  Rosters 
Session  Number  not  on  Administration 
Schedules 

Administration  Codes  not  on  A/S;  but  on 
booklets 

Administration  Codes  not  on  booklets;  but  on 
A/S 

Items  returned  for  Westat 
Writing  on  booklet  covers 
Other 


Problem  Alerts 

Change  of  Administration  Codes-A/S  or  Booklets 

Incorrect  Rosters/Questionnaires 
Administration  Notes/Writing  on  Covers 

Duplicate  Student  / Booklet  Number/ 
Administration  Schedule 
All  material  not  returned 

Affected  Testing  - Problem 
Transcribed  page(s)  for  student  booklet(s) 
Processed  as  is 

Involves  Inclusion  Check  List 

Other  _____ 


A/S  = Administration  Schedules 
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CREATION  OF  THE  DATABASE,  QUALITY  CONTROL  OF  DATA 
ENTRY,  AND  CREATION  OF  THE  DATABASE  PRODUCTS1 

John  J.  Ferris,  Katharine  E.  Pashley,  Patricia  E.  O’Reilly, 

David  S.  Freund,  and  Alfred  M.  Rogers 
Educational  Testing  Service 


6.1  OVERVIEW 

The  data  processing,  scoring,  and  editing  procedures  described  in  Chapter  5 resulted  in 
the  generation  of  disk  and  tape  files  containing  various  data  for  students  (assessed  and  excluded), 
teachers,  arid  schools,  along  with  SD/LEP  (students  with  disabilities  and  students  with  limited 
English  proficiency)  information.  The  weighting  procedures  described  in  Chapter  7 resulted  in 
the  generation  of  data  files  that  included  the  sampling  weights  required  to  make  valid  statistical 
inferences  about  the  populations  from  which  the  1996  fourth-  and  eighth-grade  State  Assessment 
mathematics  samples  were  drawn.  These  files  were  merged  into  a comprehensive,  integrated 
database.  The  creation  of  this  database  is  described  in  Section  6.2 

Section  6.3  describes  a central  repository  or  master  catalog  of  this  information.  The 
master  catalog  is  accessible  by  all  analysis  and  reporting  programs  and  provides  correct 
parameters  for  processing  the  data  fields  and  consistent  labeling  for  identifying  the  results  of  the 
analyses. 

To  evaluate  the  effectiveness  of  the  quality  control  of  the  data  entry  process,  the 
corresponding  portion  of  the  final  integrated  database  was  verified  in  detail  against  a sample  of 
the  original  instruments  received  from  the  field.  The  results  of  this  procedure  are  given  in 
Section  6.4. 


The  integrated  database  was  the  source  for  the  creation  of  the  NAEP  item  information 
database  and  the  NAEP  secondary-use  data  files.  These  are  described  in  Section  6.5. 


6.2  MERGING  FILES  INTO  THE  STATE  ASSESSMENT  DATABASE 


The  data  processing  conducted  by  National  Computer  Systems  (NCS)  resulted  in  the 
transmittal  to  ETS  of  four  data  files  for  both  fourth  and  eighth  grade:  one  for  the  student 
background  and  item  response  data  and  one  file  for  each  of  the  three  questionnaires 
(Mathematics  Teacher  Questionnaire.  School  Characteristics  and  Policies  Questionnaire,  and 
SD/LEP  Questionnaire).  The  sampling  weights,  derived  by  Westat,  Inc.,  comprised  an  additional 
seven  files  f or  each  grade — -three  sets  for  assessed  students,  three  sets  for  excluded  students  and 
one  foi  schools.  (See  Chapter  7 for  a discussion  of  the  sampling  weights.)  Thes  1 1 files  at  each 
grade  were  the  foundation  for  the  analysis  of  the  1996  State  Assessment  data.  Before  data 

1 John  .1  I ctn^  was  responsible  lot  lire  evaluation  ol  the  ijtiiilils  ot  the  database  utul  the  dute  cntr\  process,  Katharine  I.. 
l\tshlc\  was  responsible  for  database  generation  under  the  supervision  ol  David  S I ieurv!,  Patricia  l*..  O’Reilly  and 
Allied  M Ropers  created  the  secondary  use  data  Ides. 
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analyses  could  be  performed,  these  data  files  had  to  be  integrated  into  a coherent  and 
comprehensive  database. 


The  1996  State  Assessment  database  for  both  fourth  and  eighth  grade  consisted  of  two 
files — student  and  school.  Each  record  on  the  student  file  contained  a student’s  responses  to  the 
particular  assessment  booklet  the  student  was  administered — Booklets  101  to  126  (in  the  case  of 
excluded  students,  a booklet  was  assigned  but  the  student  response  fields  contain  a special  code 
indicating  no  response),  and  the  information  from  the  questionnaire  that  the  student’s 
mathematics  teacher  completed.  Additionally,  for  a student  (assessed  or  excluded)  who  was 
identified  as  SD  or  LEP,  the  data  from  the  SD/LEP  Questionnaire  is  included.  This  questionnaire 
is  filled  out  for  all  students  identified  as  SD  and/or  LEP,  both  assessed  and  excluded.  (See 
Chapter  2 for  information  regarding  assessment  instruments.)  Also  added  to  the  student  files 
were  variables  with  school-level  information  supplied  by  Quality  Education  Data,  Inc.  (QED), 
including  demographic  information  about  schools  such  as  race/ethnicity  percentages.  Since  the 
teacher  data  is  not  a representative  sample  of  teachers  and  as  the  focus  of  NAEP  is  to  report 
student  level  results,  the  teacher  response  data  was  added  to  the  student  records.  The  school  files 
were  separate  files  that  could  be  analyzed  on  their  own  and  could  also  be  linked  to  the  student 
files  through  the  unique  school  ID  code. 

The  creation  of  the  student  data  files  for  fourth  and  eighth  grade  began  with  the 
reorganization  of  the  data  files  received  from  NCS.  This  involved  two  major  tasks:  1)  the  files 
were  restructured,  eliminating  unused  (blank)  areas  to  reduce  the  size  of  the  files;  and  2)  in  cases 
where  students  had  chosen  not  to  respond  to  an  item,  the  missing  responses  were  recoded  as 
either  “omitted”  or  “not  reached,”  as  discussed  in  Chapter  9.  Next,  the  student  response  data 
were  merged  with  the  student  weights  files.  The  resulting  file  was  then  merged  with  the  SD/LEP 
and  teacher  data.  In  all  merging  steps,  the  10-digit  booklet  ID  (the  three-digit  booklet  number 
common  to  every  booklet  with  the  same  block  of  items,  a six-digit  serial  number  unique  to  the 
booklet  a student  was  given,  and  a single  check  digit,  distinguishing  bilingual  booklets)  was  used 
as  the  matching  criterion. 

The  school  file  for  each  grade  was  created  by  merging  the  School  Characteristics  and 
Policies  Questionnaire  file  with  the  file  of  school  weights  and  school  variables,  supplied  by 
Westat.  The  state  and  school  codes  were  used  as  the  matching  criteria.  Since  some  schools  did 
not  return  a questionnaire,  some  of  the  records  in  the  school  file  contained  only 
school-identifying  information  and  sampling  weight  information. 

When  the  student  and  school  files  for  each  grade  had  been  created,  the  database  was 
ready  for  analysis.  In  addition,  whenever  new  data  values,  such  as  composite  background 
variables  or  scale  scores,  were  derived,  they  were  added  to  the  appropriate  database  files  using 
the  same  matching  procedures  described  above. 

For  archival  purposes  and  to  provide  data  to  the  states,  to  researchers,  and  to 
policymakers,  secondary-use  data  files  and  codebooks  for  each  jurisdiction  were  generated  from 
this  database.  The  secondary-use  data  files,  described  in  Section  6.5.2,  contain  all  responses  and 
response-related  data  from  the  assessment,  including  responses  from  the  student  booklets. 
Teacher  Questionnaires,  and  School  Characteristics  and  Policies  Questionnaires,  scale  scores, 
sampling  weights,  and  variables  used  to  compute  standard  errors. 
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6.3  CREATING  THE  MASTER  CATALOG 


A critical  part  of  any  database  is  its  processing  control  and  descriptive  information. 
Having  a central  repository  of  this  information,  which  may  be  accessed  by  all  analysis  and 
reporting  programs,  will  provide  correct  parameters  for  processing  the  data  fields  and  consistent 
labeling  for  identifying  the  results  of  the  analyses.  The  State  Assessment  master  catalog  file  was 
designed  and  constructed  to  serve  these  purposes  for  the  State  Assessment  database. 

Each  record  of  the  master  catalog  contains  the  processing,  labeling,  classification,  and 
location  information  for  a data  field  in  the  State  Assessment  database.  The  control  parameters  are 
used  by  the  access  routines  in  the  analysis  programs  to  define  the  manner  in  which  the  data 
values  are  to  be  transformed  and  processed. 

Each  data  field  has  a 50-character  label  in  the  master  catalog  describing  the  contents  of 
the  field  and,  where  applicable,  the  source  of  the  field.  The  data  fields  with  discrete  or 
categorical  response  values  (e.g.,  multiple-choice  items,  professionally  scored  items,  and  most 
questionnaire  items,  but  not  weight  fields)  have  additional  label  fields  in  the  catalog  containing 
8-  and  20-character  labels  for  those  response  values.  These  shorter  labels  can  be  used  for 
reporting  purposes  as  a concise  description  of  the  responses  for  the  items. 

The  classification  area  of  the  master  catalog  record  contains  distinct  fields  corresponding 
to  predefined  classification  categories  (e.g.,  mathematics  content  and  process  areas)  for  the  data 
fields.  For  a particular  classification  field,  a nonblank  value  indicates  the  code  of  the  subcategory 
within  the  classification  categories  for  the  data  field.  This  classification  area  permits  the 
grouping  of  identically  classified  items  or  data  fields  by  performing  a selection  process  on  one  or 
more  classification  fields  in  the  master  catalog. 

The  master  catalog  file  was  constructed  concurrently  with  the  collection  and 
transcription  of  the  State  Assessment  data  so  that  it  would  be  ready  for  use  by  analysis  programs 
when  the  database  was  created.  As  new  data  fields  were  derived  and  added  to  the  database,  their 
corresponding  descriptive  and  control  information  were  entered  into  the  master  catalog. 
Machine-readable  catalog  files,  created  from  the  master  catalog,  are  available  as  part  of  the 
secondary-use  data  files  package  for  use  in  analyzing  the  data  with  programming  languages  other 
than  SAS  or  SPSS  (see  Section  6. 5.2. 8).  For  SAS  and  SPSS  users,  files  of  control  statements  that 
create  SAS  or  SPSS  system  files  are  provided  (see  Section  6. 5. 2. 7). 


6.4  QUALITY  CONTROL  EVALUATION 

The  purpose  of  the  data  entry  quality  control  procedure  is  to  gauge  the  overall  accuracy 
of  the  process  that  transforms  responses  into  machine-readable  data.  The  procedure  involves 
examining  the  actual  responses  made  in  a random  sample  of  booklets  and  comparing  them,  mark 
by  mark  and  character  by  character,  with  the  responses  recorded  in  the  final  database,  which  is 
used  for  analysis  and  reporting.  Notwithstanding  the  marks  made  by  the  respondent,  if  the 
respondent’s  intention  is  unambiguous,  and  if  the  data  entry'  system  has  tailed  to  accurately 
capture  the  intended  response,  the  erroneous  data  is  considered  a failure  for  purposes  of  this 
quality  control  evaluation. 
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The  selection  of  booklets  for  this  comparison  took  place  at  the  point  of  first  entry  into 
the  scanning  process  for  data  from  the  field.  These  selected  quality  control  booklets  were  set 
aside  in  a predetermined  proportion,  using  systematic  random  sampling,  and  then  collected  for 
subsequent  close  scrutiny.  Selection  proportions  comparable  to,  or  greater  than,  those  used  in 
previous  assessments  were  used.  The  results  of  this  process  are  discussed  in  detail  below,  and 
Table  6-2  contains  detailed  information  about  the  sampling  rates,  numbers  of  booklets  and  data 
characters  examined,  and  errors  found. 


6.4.1  Student  Data 

Twenty-six  assessment  booklets,  numbered  101  through  126,  were  administered  to 
students  as  part  of  the  State  Assessment  in  mathematics.  Table  6-1  provides  the  numbers  of  each 
booklet  in  the  database  for  each  grade.  Note  that  these  numbers,  and  others  reported  below  for 
various  categories  of  data,  may  vary  somewhat  from  other  totals  given  in  this  report  for  a variety 
of  reasons,  having  to  do  with  the  appropriateness  of  inclusion  for  different  purposes.  The 
variation  in  the  numbers  of  student  booklets  is  insignificant,  according  to  a chi-square  test, 
indicating  very  good  control  of  the  distribution  process. 

Student  booklets  were  sampled  in  adequate  numbers  and  the  average  rate  of  selection 
was  about  1 out  of  380,  a selection  rate  comparable  to  that  used  in  past  assessments  at  both  the 
state  and  national  levels.  The  few  errors  found  during  this  quality  control  examination  did  not 
cluster  by  booklet  number,  so  there  is  no  reason  to  believe  that  the  variation  in  numbers  of 
booklets  selected  had  a significant  effect  on  the  estimates  of  overall  error  rate  confidence  limits 
reported  below. 

The  quality  control  evaluation  detected  32  errors  in  these  student  booklet  samples,  15  at 
grade  4 and  17  at  grade  8.  All  the  errors  involved  either  multiple  responses  that  were  not 
identified  as  such  by  the  scanner  or  erasures  that  were  recorded  instead  of  ignored.  To  be 
considered  a scanning  error,  the  scanning  process  must  have  failed  to  correctly  determine  the 
respondent’s  intent  when  it  was  plain  to  the  human  eye.  While  such  a failure  might  seem  to  cast 
doubt  on  the  scanning  process,  the  final  error  rate  determined  from  the  quality  control  evaluation 
was  reassuring.  A very  large  volume  of  data  was  scanned  with  consistently  usable  results.  An 
analysis  of  this  evaluation  based  on  the  binomial  theorem  permits  the  inference  of  confidence 
limits  indicated  in  the  last  column  of  Table  6-2;  it  is  unlikely,  for  instance,  that  more  than  a tenth 
of  a percent  (.0010)  of  the  data  characters  processed  at  grade  4 would  differ  from  what  a careful 
reader  would  have  found  in  the  student  booklets. 


6.4.2  Mathematics  Teacher  Questionnaires 

A total  of  15,456  questionnaires  from  mathematics  teachers  were  associated  with 
corresponding  student  data  in  the  final  database  at  grade  4,  and  1 1 ,423  at  grade  8.  These  teacher 
questionnaires  were  sampled  at  the  rate  of  about  1 in  100,  about  the  same  rate  used  in  the 
previous  assessment.  The  265  questionnaires  selected  for  quality  control  contained  a total  of  177 
errors  in  84  different  booklets.  This  is  about  two  to  three  times  the  error  rate  found  in  recent 
assessments  for  this  instrument. 
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While  this  error  rate  is  not  bad  enough  to  render  the  teacher  data  unusable,  there  is  some 
cause  for  concern  in  the  fact  that  about  one  third  of  these  errors  were  concentrated  in  a single 
item — the  item  requesting  information  on  classroom  size.  This  suggests  the  possible  need  for 
changing  the  presentation  of  this  item  in  the  questionnaire  booklet.  While  such  changes  can 
cause  unforeseen  problems  with  response  behavior  and  data  reliability,  it  is  being  considered  in 
this  case  because  the  design  of  this  item  appears  to  be  unnecessarily  complex  and  confusing  to 
quite  a few  of  the  teachers.  Secondary  users  of  NAEP  data  are  cautioned  that  although  the 
classroom  size  was  included  as  a conditioning  variable  in  the  analysis  of  these  data,  the  errors 
found  in  the  responses  to  this  question  make  its  use  inadvisable.  Other  classroom  data  items  also 
proved  to  be  problematic  for  teachers,  especially  at  Grade  8.  The  issue  is  currently  undergoing 
study  and  alternatives  are  being  considered. 


Table  6-1 

Number  of  Mathematics  Booklets  Scanned  into  Database 
and  Selected  for  Quality  Control  Evaluation 


Booklet 

Number 

Number  of 
Booklets  in  Database 
Grade  4 Grade  8 

Number  of 
Booklets  Selected 
Grade  4 Grade  8 

101 

4,874 

4,641 

13 

10 

102 

4,827 

4,602 

13 

13 

103 

4,855 

4,641 

12 

12 

104 

4,820 

4,643 

12 

12 

105 

4,878 

4,644 

13 

10 

106 

4,858 

4,659 

13 

12 

107 

4,886 

4,684 

13 

13 

108 

4,877 

4,637 

13 

12 

109 

4,844 

4,654 

14 

10 

110 

4,877 

4,643 

12 

12 

111 

4,849 

4,613 

12 

11 

112 

4,825 

4,654 

11 

12 

in 

4,889 

4,589 

13 

10 

114 

4,892 

4,644 

13 

11 

115 

4,870 

4,614 

12 

12 

116 

4,838 

4,601 

14 

12 

117 

4,832 

4,649 

13 

13 

118 

4,836 

4,659 

13 

11 

119 

4,907 

4,570 

14 

1 1 

120 

4,879 

4,643 

13 

12 

121 

4,847 

4,589 

13 

12 

122 

4,871 

4,596 

13 

10 

123 

4,800 

4,634 

12 

12 

124 

4,867 

4,647 

13 

11 

125 

4,866 

4,639 

12 

12 

126 

4.823 

4,587 

13 

11 

Total 

126,287 

120,369 

332 

299 

BEST  COPY  AVAILABLE 
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Table  6-2 

Summary  of  the  Quality  Control  Evaluation  of  Mathematics  Data 


Subsample 

Selection 

Rate 

Different 

Booklets 

Selected 

Number 

of 

Booklets 

Selected 

Number  of 
Characters 
of  Data 

Number 

of 

Errors 

Observed 

Error 

Rate 

Upper 

99.8% 

Confidence 

Limit 

GRADE  4: 

Student 

1/380 

26 

332 

30,214 

15 

.0005 

.0010 

Teacher 

1/103 

1 

150 

24,450 

73 

.0030 

.0042 

School 

1/75 

1 

61 

10,858 

11 

.0010 

.0023 

SD/LEP 

1/100 

1 

175 

22,050 

16 

.0007 

.0015 

GRADE  8: 

Student 

1/377 

26 

299 

36,136 

17 

.0005 

.0009 

Teacher 

1/99 

1 

115 

18,515 

104 

.0056 

.0070 

School 

1/78 

1 

54 

9,882 

6 

.0006 

.0017 

SD/LEP 

1/103 

1 

123 

15,498 

17 

.0011 

.0021 

6.4.3  School  Characteristics  and  Policies  Questionnaires 

A total  of  4,595  questionnaires  were  collected  from  school  administrators  and  included 
in  the  database  at  grade  4,  and  4,225  at  grade  8.  These  questionnaires  were  sampled  for  quality 
control  evaluation  at  the  rate  of  about  1 in  75,  resulting  in  the  selection  of  61  questionnaires  at 
grade  4,  and  54  at  grade  8.  The  17  errors  that  were  found  represent  an  error  rate  about  the  same 
as  that  for  school  questionnaires  in  past  assessments,  well  below  any  reasonable  threshold  for 
alarm. 


6.4.4  SD/LEP  Student  Questionnaires 

A total  of  17,554  SD/LEP  questionnaires  were  scanned  and  included  in  the  database  at 
grade  4,  and  12,708  at  grade  8.  Nearly  half  of  these  questionnaires  represented  students  who 
were  part  of  the  cognitive  assessment;  the  balance  of  the  questionnaires  came  from  students  who 
were  excluded.  The  overall  selection  rate  was  about  1 in  100,  roughly  double  that  used  in  earlier 
assessments  for  this  type  of  questionnaire.  A total  of  298  questionnaires  were  selected  across 
both  grades.  The  resulting  error  rate  indicated  that  the  quality  of  this  data  was  second  only  to  the 
student  data  and  certainly  adequate  for  the  purposes  to  which  it  was  put. 

The  results  of  the  evaluation  of  all  questionnaire  data,  as  well  as  the  student  data,  are 
summarized  in  Table  6-2. 


6.5  NAEP  DATABASE  PRODUCTS 

The  NAEP  database  described  to  this  point  serves  primarily  to  support  analysis  and 
reporting  activities  that  are  directly  related  to  the  NAEP  cooperative  agreement.  This  database 
has  a singular  structure  and  access  methodology  that  is  integrated  with  the  NAEP  analysis  and 


reporting  programs.  One  of  the  directives  of  the  NAEP  cooperative  agreement  is  to  provide 
secondary  researchers  with  a nonproprietary  version  of  the  database  that  is  portable  to  any 
computer  system.  In  the  event  of  transfer  of  NAEP  to  another  client,  the  cooperative  agreement 
further  requires  ETS  to  provide  a full  copy  of  the  internal  database  in  a format  that  may  be 
installed  on  a different  computer  system. 

In  fulfillment  of  these  requirements,  ETS  provides  two  sets  of  database  products:  the 
item  information  database  and  the  secondary-use  data  files.  The  contents,  format,  and  usage  of 
these  products  are  documented  more  extensively  in  the  publications  listed  under  the  appropriate 
sections  below. 


6.5- 1 The  Item  Information  Database 

The  NAEP  item  information  database  contains  all  of  the  descriptive,  processing,  and 
usage  information  for  each  item  or  variable  used  for  NAEP  since  1970.  The  primary  unit  of  this 
database  is  the  item.  Each  NAEP  item  is  associated  with  different  levels  of  information, 
including  usage  across  years  and  age  cohorts,  subject  area  classifications,  response  category 
descriptors,  and  locations  of  response  data  on  secondary-use  data  files. 

The  item  information  database  can  be  used  for  a variety  of  NAEP  tasks:  providing 
statistical  information  to  aid  in  test  construction,  determining  the  usage  of  items  across 
assessment  years  and  ages  for  trend  and  cross-sectional  analyses,  providing  text  labels  for 
analyses  and  reports,  and  organizing  items  by  subject  area  classifications  for  scaling  analysis. 


6,5.2  The  Secondary-Use  Data  Files 

The  secondary-use  data  files  are  designed  to  enable  any  researcher  with  an  interest  in 
NAEP  to  perform  secondary  analysis  on  the  same  data  as  those  used  for  analysis  at  ETS. 
Supporting  documentation  accompanies  the  data  files.  The  set  of  files  for  each  sample  (e.g.,  the 
North  Dakota  grade  4 assessed  students)  or  instrument  (e.g.,  the  Florida  grade  4 school  data) 
includes:  a file  containing  the  data;  a file  of  control  statements  that  will  generate  an  SPSS  system 
file;  a file  of  control  statements  that  will  generate  a SAS  system  file;  and  a machine-readable 
catalog  file.  Each  machine-readable  catalog  file  (discussed  in  Section  6. 5. 2. 8)  contains  sufficient 
control  and  descriptive  information  to  aid  those  users  without  SAS  or  SPSS  to  set  up  and  perform 
data  analyses.  The  printed  documentation  consists  of  three  volumes:  a guide  to  the  use  of  the  data 
files,  and  for  each  grade,  a set  of  data  file  layouts  and  codebooks  for  each  participating 
jurisdiction. 

The  remainder  of  this  section  summarizes  the  procedures  used  in  generating  the  data  files 
and  related  materials.  More  information  about  the  contents  and  use  of  the  data  files  is  contained 
in  the  NAEP  1996  State  Assessment  Program  in  Mathematics  Secondary- Use  Data  Files  User 
Guide  (O’Reilly,  Zelenak,  Rogers,  & Kline,  1997). 
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6.5.2. 1 File  Definition 

There  are  essentially  four  samples  for  analysis  in  the  19%  State  Assessment  in 
mathematics:  the  students  (assessed  and  excluded),  the  schools  in  the  State  Assessment,  and  the 
students  and  the  schools  in  a matched  National  Reporting  Sample  drawn  from  the  national 
mathematics  assessment.  The  four  samples  are  divided  into  separate  files  by  grade  and  also  by 
participating  jurisdiction  (for  the  two  State  Assessment  samples),  resulting  in  a total  of  over  190 
files;  however,  the  same  file  formats,  file  linking  conventions,  and  analysis  considerations  apply 
to  each  file  within  a given  sample.  For  example,  the  analysis  specification  that  links  school  and 
student  data  for  a given  grade  for  California  would  apply  identically  to  New  \ ork,  Tennessee,  or 
any  other  participating  jurisdiction  or  group  of  jurisdictions. 

Every  data  file  for  each  participating  jurisdiction  requires  its  own  data  codebook, 
detailing  the  frequencies  of  data  values  within  that  jurisdiction  for  the  given  sample  and  grade. 
The  file  layouts,  SPSS  and  SAS  syntax,  and  machine-readable  catalog  files,  however,  need  only 
be  generated  for  each  of  the  four  samples  at  grade  4 and  at  grade  8,  since  the  individual 
jurisdiction  data  files  for  each  1996  State  Assessment  sample  at  a given  grade  are  identical  in 
format  and  data  code  definition. 


6.S.2.2  Definition  of  the  Variables 

Prior  to  the  1990  assessment,  information  that  could  potentially  be  used  to  identity 
students  or  schools  was  not  included  on  the  secondary-use  files.  When  these  public-use  data  files 
were  replaced  bv  the  current  restricted-use  data  files,  the  restraint  on  confidential  data  was  lilted. 
This  change  simplified  the  variable  definition  process,  as  it  permitted  the  transfer  of  all  variables 
from  the  database  to  the  secondary-use  files. 

The  initial  step  in  this  process  was  the  generation  of  a LABELS  file  of  descriptors  ot  the 
variables  for  each  data  file  to  be  created.  Each  record  in  a LABELS  file  contains,  tor  a single 
data  field,  the  variable  name,  a short  description  of  the  variable,  and  processing  control 
information  to  be  used  by  later  steps  in  the  process  of  generating  the  secondary-use  data  files. 
ETS  staff  could  edit  this  file  for  deletion  of  variables,  modification  of  control  parameters,  or 
reordering  of  the  variables  within  the  file.  The  LABELS  tile  is  an  intermediate  file  only;  it  is  not 
distributed  with  the  secondary-use  data  tiles. 


The  next  program  in  the  processing  stream.  GENLY1,  produced  a printed  layout  tor 
each  data  file  from  the  information  in  its  corresponding  LABELS  tile.  These  layouts  are 
reviewed  for  the  ordering  of  the  variables.  The  variables  on  all  data  tiles  are  grouped  and 
arranged  in  the  following  order:  identification  information,  weights,  derived  variables,  scale 
scores  (where  applicable),  and  item  response  data.  On  the  student  data  tiles,  the'se  fields  arc 
followed  by  the  teacher  response  data  and  the  SD/LEP  student  questionnaire  data,  where 
applicable.  The  identification  information  is  taken  from  the  front  covers  of  the  instruments.  The 
weight  data  include  sample  descriptors,  selection  probabilities,  and  replicate  weights  for  the 
estimation  of  sampling  error.  The  derived  data  include  sample  descriptions  from  other  sources 
and  variables  that  are  derived  from  the  item  response  data  for  use  in  analysis  or  reporting.  Item 
response  data  consist  of  responses  to  questionnaire  items;  for  assessed  students,  these  data 
include  responses  to  cognitive  items,  as  well. 
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In  the  assessed  student  data  files  for  each  participating  jurisdiction  of  the  State 
Assessment  in  mathematics  and  for  the  National  Reporting  Sample,  the  item  response  data  within 
each  block  were  left  in  their  order  of  presentation.  The  blocks,  however,  were  arranged  according 
to  the  following  scheme:  common  background,  subject-related  background,  the  cognitive  blocks 
in  ascending  numerical  order,  and  student  motivation.  The  responses  to  cognitive  blocks  that 
were  not  present  in  a given  booklet  were  left  blank,  signifying  a condition  of  ‘missing  by  design.’ 

In  order  to  process  and  analyze  the  spiral  sample  data  effectively,  the  user  must  also  be 
able  to  determine,  from  a given  booklet  record,  which  blocks  of  item  response  data  were  present 
and  their  relative  order  in  the  instrument.  The  user  obtains  this  information  from  a set  of  control 
variables,  one  for  each  block,  which  indicate  not  only  the  presence  or  absence  of  the  block  but  its 
order  in  the  instrument.  These  control  variables  created  by  ETS  are  included  with  the  derived 
variables. 


6.S.2.3  Data  Definition 

To  enable  the  data  files  to  be  processed  on  any  computer  system  using  any  procedural  or 
programming  language,  it  was  desirable  that  the  data  be  expressed  in  numeric  format.  This  was 
possible,  but  not  without  the  adoption  of  certain  conventions  for  expressing  the  data  values 
numerically. 

During  creation  of  the  NAEP  database,  the  responses  to  all  multiple-choice  items  (both 
cognitive  multiple-choice  items  and  those  in  the  questionnaires)  were  processed  and  stored  in  the 
database  using  the  letter  codes  printed  in  the  instruments.  This  scheme  afforded  the  advantage  of 
saving  storage  space  for  items  with  10  or  more  response  options,  but  at  the  expense  of  translating 
these  codes  into  their  numeric  equivalents  for  analysis  purposes.  The  response  data  fields  for 
most  of  these  items  would  require  a simple  alphabetic-to-numeric  conversion.  However,  the  data 
fields  for  items  with  10  or  more  response  choices  would  require  “expansion”  before  the 
conversion,  since  the  numeric  value  would  require  two  column  positions.  One  of  the  processing 
control  parameters  on  the  LABELS  file  indicates  whether  or  not  the  data  field  is  to  be  expanded 
before  conversion  and  output  to  the  secondary-use  data  files. 

The  ETS  database  contained  special  codes  to  indicate  certain  response  conditions: 

"I  don't  know”  responses,  multiple  responses,  omitted  responses,  not-reached  responses,  and 
unresolvable  responses,  which  included  out-of-range  responses  and  responses  that  were  missing 
due  to  errors  in  printing  or  processing.  The  scoring  guides  for  the  mathematics  constructed- 
response  items  included  additional  special  codes  for  ratings  of  erased  or  crossed  out  and  for 
ratings  of  illegible,  “I  don’t  know,”  off  task,  or  nonratable  by  the  scorers.  All  of  these  codes  had  to 
be  reexpressed  in  a consistent  numeric  format. 

The  following  convention  was  adopted  and  used  in  the  designation  of  these  codes:  The 
“I  don't  know”  and  nonratable  response  codes  (including  off-task  and  illegible  responses)  were 
always  converted  to  7;  the  omitted  response  codes  were  converted  to  8;  the  not-reached  response 
codes  were  converted  to  9;  the  multiple  response  codes  were  converted  to  0;  and  the  erased  and 
crossed  out  response  codes  were  converted  to  5.  The  out-of-range  and  missing  responses  were 
coded  as  blank  fields,  corresponding  to  the  ‘missing  by  design’  designation. 

This  coding  scheme  created  conflicts  for  those  multiple-choice  items  that  had  seven  or 
more  valid  response  options  as  well  as  the  “I  don’t  know”  response,  and  also  for  those 
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constructed-response  items  whose  scoring  guide  had  five  or  more  categories.  These  data  fields 
were  also  expanded  to  accommodate  the  valid  response  values  and  the  special  codes.  In  these 
cases,  the  special  codes  were  ‘extended’  to  fill  the  output  data  field:  the  “I  don’t  know”  and 
nonratable  codes  were  extended  from  7 to  77,  omitted  response  codes  from  8 to  88,  etc. 

Each  numeric  variable  on  the  secondary-use  files  was  classified  as  either  continuous  or 
discrete.  These  classifications  are  related  to  machine-level  characteristics,  rather  than  to  the 
precise  mathematical  meaning  of  these  terms.  The  discrete  variables  include  those  items  for 
which  each  numeric  value  corresponds  to  a response  category.  The  continuous  variables  include 
the  weights,  scale  scores,  the  identification  information  codes,  and  questionnaire  item  responses 
for  which  counts  or  percentages  were  requested.  The  designation  of  “discrete”  includes  those 
derived  variables  to  which  numeric  classification  categories  have  been  assigned.  The 
constructed-response  items  were  treated  as  a special  subset  of  the  discrete  variables  and  were 
assigned  to  a separate  category  to  facilitate  their  identification  in  the  documentation. 


6.5.2.4  Data  File  Catalogs 

The  LABELS  file  contains  sufficient  descriptive  information  for  generating  a brief 
layout  of  the  data  file.  However,  to  generate  a complete  codebook  document,  substantially  more 
information  about  the  data  is  required.  The  CATALOG  file  provides  most  of  this  information. 

The  CATALOG  file  is  created  by  the  GENCAT  program  from  the  LABELS  file  and  the 
1996  master  catalog  file,  as  described  in  Section  6.3.  Each  record  on  the  LABELS  file  generates 
a CATALOG  record  by  first  retrieving  the  master  catalog  record  corresponding  to  the  field  name. 
The  master  catalog  record  contains  usage,  classification,  and  response  code  information,  along 
with  positional  information  from  the  LABELS  file:  field  sequence  number,  output  column 
position,  and  field  width.  Like  the  LABELS  file,  the  CATALOG  file  is  an  intermediate  file  and 
is  not  included  with  the  secondary-use  data  files. 

The  information  for  the  response  codes,  also  referred  to  as  “foils,”  consists  of  the  valid 
data  values  for  the  discrete  numeric  fields  and  a 20-character  descriptive  label  for  each  valid  data 
value.  (Readers  who  are  familiar  with  standard  usage  of  the  term  “foil”  in  testing  and 
measurement  will  notice  that  it  has  an  expanded  meaning  in  this  discussion  of  the  secondary-use 
data  files.)  The  GENCAT  program  uses  additional  control  information  from  the  LABELS  file  to 
determine  if  extra  foils  should  be  generated  and  saved  with  each  CATALOG  record.  The  first 
flag  controls  generation  of  the  “I  don’t  know”  or  nonratable  foil;  the  second  flag  regulates 
omitted  or  not-reached  foil  generation;  and  the  third  flag  denotes  the  possibility  of  multiple 
responses  for  that  field  and  sets  up  an  appropriate  foil.  All  of  these  control  parameters,  including 
the  expansion  flag,  may  be  altered  in  the  LABELS  file  by  use  of  a text  editor,  in  order  to  control 
the  generation  of  data  or  descriptive  information  for  any  given  field. 

The  LABELS  file  supplies  control  information  for  many  of  the  subsequent 
secondary-use  data  processing  steps.  The  CATALOG  file  provides  detailed  information  for  those 
and  other  steps. 
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6.5.2.S  Data  File  Layouts 


The  data  file  layouts,  as  mentioned  above,  were  the  first  user  product  to  be  generated  in 
the  secondary-use  data  files  process.  The  generation  program,  GENLYT,  used  a LABELS  file, 
described  in  Section  6.5. 2. 2,  and  a CATALOG  file  as  input  and  produced  a printable  file.  The 
LAYOUT  file  is  basically  a formatted  listing  of  the  LABELS  file;  it  documents  the  layout  and 
contents  of  the  data  files.  The  layouts  are  part  of  the  printed  documentation;  the  secondary-use 
data  file  package  includes  not  only  the  printed  layouts,  but  also  the  electronic  files  from  which 
they  were  printed. 

Each  line  of  the  LAYOUT  file  contains  the  following  information  for  a single  data  field: 
sequence  number,  field  name,  output  column  position,  field  width,  number  of  decimal  places, 
data  type,  value  range,  key  or  correct  response  value,  and  a short  description  of  the  field.  The 
sequence  number  of  each  field  is  implied  from  its  order  on  the  LABELS  file.  The  field  name  is 
an  8-character  label  for  the  field  that  is  used  consistently  by  all  secondary-use  data  file  materials 
to  refer  to  that  field  on  that  file.  The  output  column  position  is  the  relative  location  of  the 
beginning  of  that  field  on  each  record  for  that  file,  using  bytes  or  characters  as  the  unit  of 
measure.  The  field  width  indicates  the  number  of  columns  used  in  representing  the  data  values 
for  a field.  If  the  field  contains  continuous  numeric  data,  the  value  under  the  number  of  decimal 
places  entry  indicates  how  many  places  to  shift  the  decimal  point  before  processing  data  values. 

The  data  type  category  uses  five  codes  to  designate  the  nature  of  the  data  in  the  field: 
Continuous  numeric  data  are  coded  “C';  discrete  numeric  data  are  coded  “D”;  constructed- 
response  item  data  are  coded  “OS”  if  the  item  was  dichotomized  for  scaling  and  “OE”  if  it  was 
scaled  under  a polytomous  response  model.  Additionally,  the  discrete  numeric  fields  that  include 
“I  don’t  know”  response  codes  are  coded  “DI”  If  the  field  type  is  discrete  numeric,  the  value 
range  is  listed  as  the  minimum  and  maximum  permitted  values  separated  by  a hyphen  to  indicate 
range.  If  the  field  is  a response  to  a multiple-choice  item,  the  correct  option  value,  or  key,  is 
printed;  if  the  field  is  an  assigned  score  for  a constructed-response  item  that  was  scaled  as  a 
dichotomous  item  using  outpoint  scoring,  the  range  of  correct  scores  is  printed.  Each  variable  is 
further  identified  by  its  50-character  descriptive  label. 


6.5,2,6  Data  Codebooks 

The  data  codebooks  form  the  bulk  of  the  printed  documentation  of  the  secondary-use 
data  files;  they  contain  complete  descriptive  information  for  each  data  field.  Most  of  this 
information  originates  from  the  CATALOG  file;  the  remaining  data  comes  from  the  COUNTS 
file  and  the  IRT  parameters  file,  described  below.  The  secondary-use  data  file  package  includes 
the  electronic  files  from  which  the  codebooks  were  printed,  in  addition  to  the  printed  codebooks. 

Each  data  field  receives  at  least  one  line  of  descriptive  information  in  the  codebook.  If 
the  data  type  is  continuous  numeric,  no  more  information  is  given.  If  the  variable  is  discrete 
numeric,  the  eodebook  lists  the  foil  codes,  foil  labels,  and  frequencies  of  each  value  in  the  data 
file.  Additionally,  if  the  field  represents  an  item  used  in  IRT  scaling,  the  codebook  lists  the  final 
parameters  estimated  by  the  scaling  program.  (See  Chapters  8 and  9 for  information  about 
scaling.) 


Certain  blocks  of  cognitive  items  in  the  1996  assessment  that  are  to  be  used  again  in  later 
assessments  for  trend  comparisons  have  been  designated  as  nonreleased.  In  order  to  maintain 
confidentiality  of  nonreleased  multiple-choice  items,  generic  foil  labels  have  been  substituted  for 
the  foils  (i.e.,  the  response  category  descriptions)  for  these  items  in  the  data  codebooks  and  the 
secondary-use  files. 

The  frequency  counts  are  not  available  on  the  CATALOG  file,  but  must  be  generated 
from  the  data.  The  GENFREQ  program  creates  the  COUNTS  file  using  the  field  name  to  locate 
the  variable  in  the  database,  and  the  foil  values  to  validate  the  range  of  data  values  for  each  field. 
This  program  also  serves  as  a check  on  the  completeness  of  the  foils  in  the  CATALOG  file,  as  it 
flags  any  data  values  not  represented  by  a foil  value  and  label. 

The  IRT  parameter  file  is  linked  to  the  CATALOG  file  through  the  field  name.  Printing 
of  the  IRT  parameters  is  governed  by  a control  flag  in  the  classification  section  of  the 
CATALOG  record.  If  an  item  has  been  scaled,  and,  thus,  used  in  deriving  the  scale  scores,  the 
IRT  parameters  are  listed  to  the  right  of  the  foil  values  and  labels,  and  the  score  value  for  each 
response  code  is  printed  to  the  immediate  right  of  the  corresponding  frequency. 

The  LAYOUT  and  CODEBOOK  files  are  written  by  their  respective  generation 
programs  to  print-image  disk  data  files.  Draft  copies  are  printed  and  distributed  for  review  before 
the  production  copy  is  generated.  The  production  copy  is  printed  on  an  IBM  printer  that  uses 
laser-imaging  technology  to  produce  high-quality,  reproducible  documentation. 

6.5.2.7  Control  Statement  Files  for  Statistical  Packages 

An  additional  requirement  of  the  NAEP  cooperative  agreement  is  to  provide,  for  each 
secondary-use  data  file,  a file  of  SAS  control  statements  that  will  convert  the  secondary-use  data 
file  into  a system  data  file  for  use  with  the  SAS  statistical  system.  Also  required  is  a file  of  SPSS 
control  statements  that  will  produce  a system  data  file  for  the  SPSS  statistical  system.  Two 
separate  programs.  GENSAS  and  GENSPX,  generate  these  control  statement  files  using  the 
CATALOG  file  as  input. 


The  control  statement  files  create  a SAS  or  SPSS  system  data  lile  that  corresponds  to  an 
entire  NAEP  secondary-use  data  file.  NAEPEX,  the  NAEP  data  extraction  software  described  in 
Section  6. 5. 2.9,  can  be  used  to  produce  control  statement  files  that  create  a SAS  or  SPSS  system 
data  file  corresponding  to  a user-defined  subset  of  the  NAEP  secondary-use  data  files.  Also  ^ 
described  in  that  section  are  the  NAEP  analysis  modules,  currently  available  for  use  with  SPSS4 
for  Windows™. 

Each  of  the  control  statement  files  contains  separate  sections  for  variable  definition, 
variable  labeling,  missing  value  declaration,  value  labeling,  and  creation  of  scored  variables  from 
the  cognitive  items.  The  variable  definition  section  describes  the  locations  of  the  fields,  by  name, 
in  the  file,  and,  if  applicable,  the  number  of  decimal  places  or  type  of  data.  The  variable  label 
identifies  each  field  with  its  50-character  descriptive  label.  The  missing  value  section  identifies 
values  of  those  variables  that  are  to  be  treated  as  missing  and  excluded  from  analyses.  The  value 
labels  correspond  to  the  foils  in  the  CATALOG  file.  The  code  values  and  their  descriptors  are 
listed  for  each  discrete  numeric  variable.  The  scoring  section  is  provided  to  permit  secondary 
users  to  generate  item  score  variables  in  addition  to  the  item  response  variables. 


Each  of  the  control  statement  generation  programs  combines  three  steps  into  one 
complex  procedure.  As  each  CATALOG  file  record  is  read,  it  is  broken  into  several  component 
records  according  to  the  information  to  be  used  in  each  of  the  resultant  sections.  These 
component  records  are  tagged  with  the  field  sequence  number  and  a section  sequence  code.  They 
are  then  sorted  by  section  code  and  sequence  number.  Finally,  the  reorganized  information  is 
output  in  a structured  format  dictated  by  the  syntax  of  the  processing  language. 

ETS  tests  the  control  statement  files  by  using  them  to  generate  system  data  files  from  the 
secondary-use  data  files.  The  control  statement  files  are  distributed  in  the  secondary-use  data 
files  package  to  permit  users  with  access  to  SAS  and/or  SPSS  to  create  their  own  system  data 
files. 


6.5.2.8  Machine-Readable  Catalog  Files 

For  those  NAEP  data  users  who  have  neither  SAS  nor  SPSS  capabilities,  yet  require 
processing  control  information  in  a computer-readable  format,  the  distribution  files  also  contain 
machine-readable  catalog  files.  Each  machine-readable  catalog  record  contains  processing 
control  information,  IRT  parameters,  and  foil  codes  and  labels. 


6.5.2.9  Secondary-Use  Data  Files  on  CD-ROM 

The  complete  set  of  secondary-use  data  files  described  above  are  available  on  CD-ROM 
as  part  of  the  NAEP  Data  on  Disk  product  suite.  This  medium  can  be  used  by  researchers  and 
policy  makers  operating  in  a personal  computing  environment. 

The  NAEP  Data  on  Disk  product  suite  includes  two  additional  components  that  facilitate 
the  analysis  of  NAEP  secondary-use  data.  The  PC-based  NAEP  data  extraction  software, 
NAEPEX,  enables  users  to  create  customized  extracts  from  the  NAEP  secondary-use  data  files 
and  to  generate  SAS  or  SPSS  control  statements  for  preparing  analyses  or  generating  customized 
system  files.  Both  Windows  3.1  and  DOS  versions  of  NAEPEX  are  available.  The  NAEP 
analysis  modules,  which  currently  run  under  SPSS®  for  Windows™,  use  output  files  from  the 
extraction  software  to  perform  analyses  that  incorporate  statistical  procedures  appropriate  for  the 
NAEP  design. 

Summarized  NAEP  data  in  tabular  format  (the  NAEP  data  almanacs  described  in 
Chapter  10)  are  also  available  on  CD-ROM.  This  product,  which  is  distinct  from  the  sece  ldary- 
use  data  files,  includes  the  NAEP  almanac  viewer,  a program  that  allows  users  to  locate  and 
display  data  of  interest. 
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Chapter  7 


WEIGHTING  PROCEDURES  AND  VARIANCE  ESTIMATION1 

John  Burke  and  Penny  James 
Westat,  Inc . 

7.1  OVERVIEW 

Following  the  collection  of  assessment  and  background  data  from  and  about  assessed  and 
excluded  students,  the  processes  of  deriving  sampling  weights  and  associated  sets  of  replicate 
weights  were  carried  out.  The  sampling  weights  are  needed  to  make  valid  inferences  from  the 
student  samples  to  the  respective  populations  from  which  they  were  drawn.  Replicate  weights  are 
used  in  the  estimation  of  sampling  variance,  through  the  procedure  known  as  jackknife  repeated 
replication. 

Each  student  was  assigned  a weight  to  be  used  for  making  inferences  about  the  state’s 
students.  This  weight  is  known  as  the  full-sample  or  overall  sample  weight.  The  full-sample 
weight  contains  three  components.  First,  a base  weight  is  established  that  is  the  inverse  of  the 
overall  probability  of  selection  of  the  sampled  student.  The  base  weight  incorporates  the 
probability  of  selecting  a school  and  the  student  within  a school.  This  weight  is  then  adjusted  for 
two  sources  of  nonparticipation — school  level  and  student  level.  These  weighting  adjustments 
seek  to  reduce  the  potential  for  bias  from  such  nonparticipation  by  increasing  the  weights  of 
students  from  schools  similar  to  those  schools  not  participating,  and  increasing  the  weights  of 
students  similar  to  those  students  from  within  participating  schools  who  did  not  attend  the 
assessment  session  (or  makeup  session)  as  scheduled.  The  details  of  how  these  weighting  steps 
were  implemented  are  given  in  Sections  7.2  and  7.3. 

Section  7.4  addresses  the  effectiveness  of  the  adjustments  made  to  the  weights  using  the 
procedures  described  in  Section  7.3.  The  section  examines  characteristics  of  nonresponding 
schools  and  students,  and  investigates  the  extent  to  which  nonrespondents  differ  from 
respondents  in  ways  not  accounted  for  in  the  weight  adjustment  procedures.  Section  7.5 
considers  the  distributions  of  the  final  student  weights  in  each  jurisdiction,  and  whether  there 
were  outliers  that  called  for  further  adjustment. 

In  addition  to  the  full-sample  weights,  a set  of  replicate  weights  was  provided  for  each 
student.  These  replicate  weights  are  used  in  calculating  the  sampling  errors  of  estimates  obtained 
from  the  data,  using  the  jackknife  repeated  replication  method.  Full  details  of  the  method  of 
using  these  replicate  weights  to  estimate  sampling  errors  are  contained  in  the  Technical  Report  of 
the  NAEP  1994  Trial  State  Assessment  Program  in  Reading  (Mazzeo,  Allen,  & Kline,  1995)  and 
in  earlier  state  technical  reports.  Section  7.6  of  this  report  describes  how  the  sets  of  replicate 
weights  were  generated  for  the  1996  State  Assessment  data.  The  methods  of  deriving  these 
weights  were  aimed  at  reflecting  the  features  of  the  sample  design  appropriately  in  each 
jurisdiction,  so  that  when  the  jackknife  variance  estimation  procedure  is  implemented, 
approximately  unbiased  estimates  of  sampling  variance  result. 


1 In  addition  to  his  responsibility  in  the  sampling  activities.  John  Burke  was  responsible  for  directing  all  weighting  and 
variance  estimation  procedures.  Penny  James  contributed  by  carrying  out  most  of  these  procedures. 
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As  detailed  in  Chapter  4,  two  different  sets  of  inclusion  rules  indicated  by  the  sample 
type  field  were  used  in  the  i 996  State  Assessment  program.  To  enable  ETS  to  analyze  these 
subsets  separately,  the  student  weights  for  each  subset  were  raked  in  order  to  force  agreement 
with  the  totals  estimated  using  both  subsets  combined.  This  raking  process  is  detailed  in 
Section  7.7. 


7.2  CALCULATION  OF  BASE  WEIGHTS 
7.2.1  Calculation  of  School  Base  Weights 

The  base  weight  assigned  to  a school  wf'h  was  the  reciprocal  of  the  probability  of 
seleuion  of  that  school.  For  the  eighth-grade  samples  and  fourth-grade  Department  of  Defense 
Domestic  Dependent  Elementary  and  Secondary  Schools  (DDESS)  and  Department  of  Defense 
Dependents  Schools  (DoDDS),  the  school  base  weight  depended  on  the  subject  of  assessment 
because  some  schools  were  so  small  that  students  were  tested  in  only  one  subject.  For  “new” 
schools  selected  using  the  supplemental  new  school  sampling  procedures  (see  Chapter  3),  the 
school  base  weight  reflected  the  combined  probability  of  selection  of  the  district,  and  school 
within  district. 

In  each  jurisdiction,  all  schools  included  in  the  sample  with  certainty  were  assigned 
school  base  weights  of  unity.  Schools  sampled  with  certainty  were  sometimes  selected  more  than 
once  in  the  systematic  sampling  process.  For  example,  a school  that  was  selected  twice  was 
allocated  twice  the  usual  number  of  students  for  the  assessments,  or  two  sessions;  a school  that 
was  selected  three  times  was  allocated  three  times  the  usual  number  of  students  for  the 
assessments,  or  three  sessions.  All  schools  at  grade  8 and  DDESS  and  DoDDS  schools  at  grade  4 
that  had  less  than  20  students  were  assigned  one  subject  (See  Chapter  3).  For  these  schools,  the 
base  weight  included  a factor  of  2.  Additional  details  about  the  weighting  process  are  given  in 
the  sections  below. 


7.2.2  Weighting  New  Schools 

New  public  schools  were  identified  and  sampled  through  a two-stage  sampling  process, 
involving  the  selection  of  districts,  and  then  of  new  schools  within  selected  districts.  This 
process  is  described  in  Chapter  3.  There  were  two  distinct  processes  used  depending  upon  the 
size  of  the  district. 

Within  each  jurisdiction,  public  school  districts  were  partitioned  into  “small"  districts, 
which  are  those  having  at  most  three  schools  on  the  aggregate  frame  and  no  more  than  one 
fourth-,  one  eighth-,  and  one  twelfth-grade  school.  The  remainder  of  the  districts  were  denoted  as 
“large”  districts.  For  the  larger  districts  (those  having  multiple  schools  in  at  least  one  of  grades  4, 
8,  and  12),  a sample  of  districts  was  selected  in  each  jurisdiction.  Districts  in  the  sample  were 
asked  to  identify  schools  having  grade  4 or  grade  8 that  were  not  included  on  the  school  frame.  A 
sample  of  these  newly  identified  schools  was  then  selected.  The  base  weight  for  these  schools 
reflected  both  the  probability  that  the  district  was  selected  for  this  updating  process,  and  that  the 
school  was  included  in  the  NAEP  sample,  having  been  identified  as  new  by  the  district.  If  the 
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school  was  in  grade  8 or  grade  4 DDHSS  and  DoDDS  schools,  but  was  only  large  enougn  to  do 
one  subject,  the  base  weight  included  a factor  of  2 as  described  in  Section  7.2.1 . 

There  were  no  schools  identified  in  small  districts  (see  Tables  3-9  and  3-10). 


7,2.3  Treatment  of  Substitute  and  Double-Session  Substitute  Schools 

Schools  that  replaced  a refusing  school  (i.e.,  substitute  schools)  were  assigned  the  weight 
of  the  refusing  school.  Thus  the  substitute  school  was  treated  as  if  it  were  the  original  school  that 
it  replaced,  for  purposes  of  obtaining  school  base  weights.  Schools  conducting  extra  sessions  that 
served  as  substitutes  for  a refusing  school  (i.e.,  double-session  substitutes)  in  effect  had  two 
school  weights.  The  students  in  the  school  who  were  assigned  to  the  original  session  were  given 
the  school  base  weight  of  the  participating  school,  while  those  students  assigned  to  the  extra 
sessionfs)  were  assigned  the  school  base  weight  of  the  refusing  school.  The  base  weight  was 
adjusted  by  a factor  of  2 if  the  grade  8 or  DDESS  or  DoDDS  school  was  only  large  enough  to  do 
one  subject. 


7.2.4  Calculation  of  Student  Base  Weights 

Within  the  sampled  schools,  eligible  students  were  sampled  for  assessment  using  the 
procedures  described  in  Chapter  3.  The  within-school  probability  of  selection  for  mathematics 
therefore  depended  on  the  number  of  grade-eligible  students  in  the  school  and  the  number  of 
students  selected  for  the  assessment  (usually  30).  The  within-school  weights  for  the  substitute 
schools  were  further  adjusted  to  compensate  for  differences  in  the  sizes  of  the  substitute  and  the 
originally  sampled  (replaced)  schools.  In  the  case  of  the  fourth  grade  DDESS  and  DoDDS 
schools  and  all  eighth-grade  schools,  the  within-school  weight  also  reflected  the  fact  that  a small 
school  could  have  been  selected  for  one  subject  but  not  the  other.  Thus,  in  general,  the 
within-school  student  weight  for  the y'th  student  in  school  i was  equal  to: 


where 


W 


within  _ N i 


Km  K2l 


tu 


N,  = the  number  of  grade-eligible  students  enrolled  in  the  school,  as  reported 
in  the  sampling  worksheets;  and 

= the  number  of  students  selected  for  the  given  subject. 

The  factors  Kh  and  K:,  in  the  formula  for  the  within-school  student  weight  generally 
apply  to  only  a few  schools  in  each  jurisdiction.  The  factor  adjusts  the  count  of  grade-eligible 
students  in  a substitute  school  be  consistent  with  the  corresponding  count  of  the  originally 
sampled  (replaced)  school.  Specifically,  for  substitute  schools, 


with 


E,  = the  grade  enrollment  of  the  originally  sampled  (replaced)  school;  and 

E"  = the  grade  enrollment  of  the  substitute  school. 

For  nonsubstitute  schools,  Kh=  1. 

The  factor  Kn,  that  was  applied  to  schools  determined  to  be  year-round  schools,  is 
defined  as: 


Kn  = 


_J 

X~Poff 


where  p„ff  is  the  percentage  of  students  enrolled  in  the  school  who  were  not  scheduled  to  attend 
school  at  the  time  of  assessment.  For  schools  that  are  not  year-round  schools  (the  great  majority), 
K2i=  1. 


The  overall  student  base  weight  for  a student  j selected  for  the  mathematics  assessment 
in  school  / was  obtained  by  multiplying  the  school  base  weight  by  the  within-school  student 
weight  and  therefore  was  computed  as: 

Wl?jise  - w?ch  WYjithin 

7.3  ADJUSTMENTS  FOR  NONRESPONSE 

As  mentioned  earlier,  the  base  weight  for  a student  was  adjusted  by  two  factors:  one  to 
adjust  for  nonparticipating  schools  for  which  no  substitute  participated,  and  one  to  adjust  for 
students  who  were  invited  to  the  assessment  but  did  not  appear  in  the  scheduled  sessions 
(original  or  makeup). 


7.3.1  Defining  Initial  School-Level  Nonresponse  Adjustment  Classes 

School-level  nonresponse  adjustment  classes  were  created  separately  for  public  and 
nonpublic  schools  within  each  jurisdiction.  For  each  set  these  classes  were  defined  as  a function 
of  their  sampling  strata,  as  follows. 

Public  Schools.  For  each  jurisdiction,  except  Guam,  the  initial  school  nonresponse 
adjustment  classes  were  formed  by  crossclassifying  the  level  of  urbanization  and  minority  status 
(see  Chapter  3 for  definitions  of  these  characteristics).  Where  there  were  no  minority  strata 
within  a particular  level  of  urbanization,  a categorized  version  of  median  household  income  was 
used.  For  this  purpose  within  each  level  of  urbanization,  public  schools  were  sorted  by  the 
median  household  income,  and  then  divided  into  three  groups  of  about  equal  size,  representing 
low,  middle,  and  high  income  areas.  In  Guam,  where  there  was  no  information  on  minority  status 
or  median  household  income,  grade  enrollment  was  used. 
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DDESS  and  DoDDS  Schools.  For  the  jurisdictions  comprised  of  DDESS  and  DoDDS 
schools,  the  initial  nonresponse  adjustment  classes  were  defined  by  other  geographic  variables. 
For  the  DDESS  schools,  the  classes  were  defined  by  military  installation  and  grouped  by  nearby 
jurisdictions.  For  DoDDS,  schools  were  grouped  by  the  regions  of  Europe  or  the  Pacific  where 
the  military  installation  was  located. 

Nonpublic  Schools.  For  each  jurisdiction  (excluding  District  of  Columbia  and  Guam 
nonpublic  schools),  initial  nonresponse  adjustment  classes  were  formed  by  crossclassifying 
school  type  (Catholic  and  non-Catholic)  and  metropolitan  status  (metro/nonmetro).  For  District 
of  Columbia  nonpublic  schools,  these  classes  were  defined  by  crossclassifying  school  type  and 
two  levels  of  estimated  grade  enrollment  (25  or  fewer  students,  versus  26  or  more  students).  For 
Guam,  initial  nonresponse  classes  for  nonpublic  schools  were  defined  by  school  type  only.  The 
District  of  Columbia  is  entirely  metropolitan,  and  Guam  is  entirely  nonmetropolitan,  so 
alternatives  were  needed  for  these  two  jurisdictions. 


7.3.2  Constructing  the  Final  Nonresponse  Adjustment  Classes 

The  objective  in  forming  the  nonresponse  adjustment  classes  is  to  create  as  many  classes 
as  possible  that  are  internally  as  homogeneous  as  possible,  but  such  that  the  resulting 
nonresponse  adjustment  factors  are  not  subject  to  large  random  variation.  Consequently,  all 
initial  nonresponse  adjustment  classes  deemed  unstable  were  collapsed  with  suitable  neighboring 
classes  so  that:  (1)  the  combined  class  contained  at  least  six  sessions,  and  (2)  the  resulting 
nonresponse  adjustment  factor  did  not  exceed  1 ,35  (in  a few  cases  a factor  in  excess  of  1 .35  was 
permitted).  These  limits  had  been  used  for  the  1994  Trial  State  Assessment.  One  change  was 
implemented  for  the  1996  State  Assessment.  When  100  percent  of  the  public  schools  in  a 
jurisdiction  responded,  no  action  was  taken  for  a public-school  adjustment  class  that  contained 
fewer  than  six  sessions.  The  same  approach  was  used  for  nonpublic  schools  where  100  percent  of 
them  participated.  Although  clearly  there  is  no  adjustment  for  school  nonresponse  in  these  cases, 
this  change  in  procedure  could  have  an  effect  on  the  final  definition  of  the  student  nonresponse 
adjustment  classes  (Section  7.3.4). 

Public  Schools.  For  these  schools,  inadequate  nonresponse  adjustment  classes  were 
reinforced  by  collapsing  adjacent  levels  of  minority  status  (or  median  household  income  level  if 
minority  information  was  missing).  In  doing  so,  metropolitan  and  nonmetropolitan  schools  were 
not  mixed.  All  DDESS  and  DoDDS  schools  cooperated,  so  no  collapsing  of  schools  was 
necessary. 

Nonpublic  Schools.  For  nonpublic  schools,  excluding  schools  in  District  of  Columbia, 
Guam,  DDESS,  and  DoDDS,  inadequate  classes  were  reinforced  by  collapsing  adjacent  levels  of 
metropolitan-area  status.  Catholic  and  non-Catholic  schools  were  kept  apart  to  the  extent 
possible,  particularly  when  the  only  requirement  to  combine  such  schools  was  as  a means  of 
reducing  the  adjustment  factors  below  1 .35.  For  schools  in  the  District  of  Columbia,  inadequate 
classes  were  collapsed  over  similar  values  of  estimated  grade  enrollment.  Catholic  and  non- 
Catholic  schools  were  kept  apart  to  the  extent  possible.  For  nonpublic  schools  in  Guam,  Catholic 
and  non-Catholic  schools  were  collapsed  together  in  order  to  form  a stable  nonresponse 
adjustment  class. 
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7.3.3  School  Nonresponse  Adjustment  Factors 


The  school-level  nonresponse  adjustment  factor  for  the  /th  school  in  the  Mh  class  was 
computed  as: 


F(\)  _ ^Cj, 

EhiShi 

ieCh 


where 


Ch  = the  subset  of  school  records  in  class  h\ 

Whib  = the  base  weight  of  the  /th  school  in  class  h\ 

Ehi  = the  grade  enrollment  for  the  /th  school  in  class  h\ 

\ l if  the  /th  school  in  adjustment  class  h participated  in  the  assessments;  and 


Shi  = 


[0  otherwise. 

Both  the  numerator  and  denominator  of  the  nonresponse  adjustment  factor  contained  only 
schools  that  were  determined  to  have  eligible  students  enrolled. 


In  the  calculation  of  the  above  nonresponse  adjustment  factors,  a school  was  said  to  have 
participated  if: 

• It  was  selected  for  the  sample  from  the  frame  or  from  the  lists  of  new 
schools  provided  by  participating  school  districts,  and  student  assessment 
data  were  obtained  from  the  school;  or 

• The  school  participated  as  a substitute  school  and  student  assessment  data 
were  obtained  (so  that  the  substitute  participated  in  place  of  the  originally 
selected  school). 


The  nonresponse-adjusted  weight  for  the  /th  school  in  class  h was  computed  as: 


Wf!J  = ■ 
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7.3.4  Student  Nonresponse  Adjustment  Classes 


The  initial  student  nonresponse  classes  for  assessed  students  were  formed  based  on 
several  variables.  The  first  of  these  was  public/nonpublic  strata.  Public/nonpublic  strata  were 
then  crossclassified  by  a variable  created  from  combining  SD/LEP  status  and  the  sample  type  for 
the  student.  SD  denotes  students  with  disabilities,  while  LEP  denotes  students  classified  as 
having  limited  English  proficiency.  Within  these  categories,  the  initial  student  nonresponse 
adjustment  classifications  were  defined  separately  depending  on  the  SD/LEP  status  of  a student. 

If  a student  was  SD  or  LEP,  then  the  class  was  formed  by  urbanicity  crossclassified  by 
student  age.  Age  was  used  to  classify  students  into  two  groups  (for  grade  4,  those  bom  in 
September  1985  or  earlier  and  those  bom  in  October  1985  or  later,  and  for  grade  8,  those  bom  in 
September  1981  or  earlier  and  those  bom  in  October  1981  or  later).  If  a student  was  neither  SD 
nor  LEP,  then  the  initial  nonresponse  adjustment  class  was  formed  by  urbanicity  crossclassified 
by  student  age  (as  defined  above),  by  the  quality  control  monitoring  status  (see  Chapter  3),  then 
finally  by  minority  status  as  collapsed  for  the  school  nonresponse.  For  the  DDESS  and  DoDDS 
schools,  the  nonresponse  adjustment  classes  for  SD  and  LEP  students  was  student  age 
crossclassified  by  the  geographic  variable  as  defined  for  the  school  nonresponse  adjustment 
classes. 


Following  creation  of  the  initial  student  nonresponse  adjustment  classifications,  all 
unstable  classes  were  identified  for  possible  collapsing  with  other  classes.  A class  was 
considered  to  be  unstable  when  either  of  the  following  conditions  was  true  for  the  given  class: 

• Number  of  responding  eligible  students  was  fewer  than  20;  or 

• Nonresponse  adjustment  factor  exceeded  1 .5. 

All  classes  deemed  unstable  in  the  previous  step  were  collapsed  with  other  classes  using 
the  following  rules: 

• Do  not  collapse  across  public  and  nonpublic; 

• Do  not  collapse  across  SD/LEP  and  non-SD/non-LEP; 

• If  within  cells  defined  by  the  crossclassification  of  public/nonpublic  and  SD- 
LEP/nonSD-nonLEP  status,  and  sample  type  within  the  SD/LEP  categories, 
all  of  the  adjustments  are  one,  no  adjustments  are  made;  and 

• Collapse  across  the  last  variable  of  the  nonresponse  adjustment  cell  only 
(i.e.,  collapse  across  geography  for  SD/LEP  students  in  Department  of 
Defense  Education  Activity  (DoDEA)  schools). 

More  collapsing  was  necessary  only  if  the  resulting  classes  had  fewer  than  15  responding 
eligible  students.  Collapsing  then  continued  within  the  successive  variables  until  the  class  size 
was  no  longer  deficient  or  until  a “set”  boundary  that  could  not  be  crossed  was  reached. 


123 


7.3.5  Student  Nonresponse  Adjustments 


As  described  above,  the  student-level  nonresponse  adjustments  for  the  assessed  students 
were  made  within  classes  defined  by  the  SD/LEP  status,  sample  type,  final  school-level 
nonresponse  adjustment  classes,  monitoring  status  of  the  school,  and  age  group  of  the  students. 
Subsequently,  in  each  jurisdiction,  the  final  student  weight  for  the  yth  student  of  the  ith  school  in 
class  k was  then  computed  as: 


w{ = wtIj 


x vv,- 


within 


x Fk 


where 

the  nonresponse-adjusted  school  weight  for  school  /; 
the  within-school  weight  for  the  yth  student  in  school  /; 
and 


W, 


dtij  _ 


within 


5X 

Fk  = v2 — — • 

Z.WijSkj 

j 

In  the  above  formulation,  the  summation  included  all  students,  y,  in  the  kih  final  (collapsed) 
nonresponse  class.  The  indicator  variable  dkj  had  a value  of  1 when  the  yth  student  in  adjustment 

class  k participated  in  the  assessment;  otherwise,  <5*,  = 0. 

For  excluded  students,  no  nonresponse  adjustment  procedures  were  applied  because 
excluded  students  were  not  required  to  complete  an  assessment.  In  effect,  all  excluded  students 
were  considered  respondents.  Weights  are  provided  for  excluded  students  so  as  to  estimate  the 
size  of  this  group  and  its  population  characteristics.  Tables  7-1  through  7-4  summarize  the 
unweighted  and  final  weighted  counts  of  assessed  and  excluded  students  in  public  and  nonpublic 
schools  for  each  jurisdiction  and  grade. 
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Table  7-1 

Unweighted  and  Final  Weighted  Counts  of  Assessed  and  Excluded  Students  by  Jurisdiction 

Grade  4 Public  Schools 


Jurisdiction 

Assessed 

Unweighted  Weighted 

Excluded 

Unweighted  Weighted 

Assessed  and  Excluded 
Unweighted  Weighted 

Alabama 

2,610 

51,861 

189 

3,716 

2,799 

55,578 

Alaska 

2,456 

9,237 

134 

454 

2,590 

9,691 

Arizona 

2,213 

53,327 

333 

7,508 

2,546 

60,835 

Arkansas 

2,090 

30,099 

142 

2,047 

2,232 

32,146 

California 

2,247 

342,180 

537 

73,235 

2,784 

415,416 

Colorado 

2,731 

46,828 

223 

3,681 

2,954 

50,508 

Connecticut 

2,679 

35,194 

284 

3,411 

2,963 

38,605 

Delaware 

2,073 

7,797 

168 

555 

2,241 

8,352 

District  of  Columbia 

2,609 

4,953 

340 

629 

2,949 

5,583 

Florida 

2,690 

153,556 

308 

16,370 

2,998 

169,926 

Georgia 

2,618 

90,729 

231 

7,599 

2,849 

98,327 

Guam 

1,497 

2,403 

175 

271 

1,672 

2,674 

Hawaii 

2,698 

13,487 

170 

817 

2,868 

14,304 

Indiana 

2,545 

65,220 

151 

3,700 

2,696 

68,920 

Iowa 

2,435 

34,536 

170 

2,370 

2,605 

36,905 

Kentucky 

2,627 

43,170 

175 

2,775 

2,802 

45,945 

Louisiana 

2,758 

56,898 

256 

5,109 

3,014 

62,006 

Maine 

2,178 

15,786 

214 

1,412 

2,392 

17,198 

Maryland 

2,547 

56,121 

223 

4,786 

2,770 

60,908 

Massachusetts 

2,628 

63,186 

269 

6,169 

2,897 

69,356 

Michigan 

2,435 

115,749 

171 

7,930 

2,606 

123,679 

Minnesota 

2,548 

58,933 

162 

3,495 

2,710 

62,428 

Mississippi 

2,767 

36,384 

164 

2,088 

2,931 

38,472 

Missouri 

2,744 

62,724 

185 

4,000 

2,929 

66,724 

Montana 

2,317 

12,533 

119 

598 

2,436 

13,131 

Nebraska 

2,816 

21,143 

175 

1,193 

2,991 

22,336 

Nevada 

2,299 

18,557 

247 

1 ,755 

2,546 

20,312 

New  Jersey 

2,017 

81,411 

137 

5,383 

2,154 

86,794 

New  Mexico 

2,566 

21,373 

322 

2,506 

2,888 

23,879 

New  York 

2,321 

195,835 

217 

17,463 

2,538 

213,299 

North  Carolina 

2,743 

85,495 

225 

6,739 

2,968 

92,234 

North  Dakota 

2,758 

8,494 

123 

332 

2,881 

8,826 

Oregon 

2,362 

37,991 

226 

3,535 

2,588 

41,526 

Pennsylvania 

2,384 

133,892 

126 

6,354 

2,510 

140,246 

Rhode  Island 

2,622 

10,663 

201 

728 

2,823 

1 1 ,392 

South  Carolina 

2,433 

46,406 

168 

3,097 

2,601 

49,502 

Tennessee 

2,574 

61,138 

162 

3,648 

2.736 

64,786 

Texas 

2,597 

262,594 

309 

29,394 

2,906 

291,988 

Utah 

2,739 

31,973 

163 

1,786 

2,902 

33,759 

Vermont 

2,213 

7,753 

174 

572 

2,387 

8,325 

Virginia 

2,659 

73,389 

234 

6,246 

2,893 

79,635 

Washington 

2,733 

68,834 

170 

4,083 

2,903 

72,918 

West  Virginia 

2,608 

20,233 

246 

1,836 

2,854 

22,069 

Wisconsin 

2,500 

59,946 

207 

4,731 

2,707 

64,677 

Wyoming 

2,889 

7,293 

123 

293 

3,012 

7,586 

Total 

112,573 

2,717,303 

9,448 

266,400 

122,021 

2,983,703 
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Table  7-2 

Unweighted  and  Final  Weighted  Counts  of  Assessed  and  Excluded  Students  by  Jurisdiction 

Grade  4 Nonpublic  Schools 


Jurisdiction 

Assessed 

Unweighted  Weighted 

Excluded 

Unweighted  Weighted 

Assessed  and  Excluded 
Unweighted  Weighted 

Alabama 

239 

5,898 

0 

0 

239 

5,898 

Arizona 

207 

5,121 

3 

99 

210 

5,219 

Arkansas 

176 

2,072 

4 

47 

180 

2.119 

California 

256 

44,677 

0 

0 

256 

44,677 

Colorado 

179 

3,670 

4 

99 

183 

3,770 

Connecticut 

255 

4,351 

3 

48 

258 

4,399 

Delaware 

338 

1,844 

2 

10 

340 

1,854 

District  of  Columbia 

410 

1,090 

2 

6 

412 

1.0% 

DoDEA/DDESS 

1 ,340 

2,866 

58 

1 17 

l .398 

2,983 

DoDEA/DoDDS 

2,670 

6,527 

143 

342 

2,813 

6,869 

Florida 

232 

16,799 

2 

1 15 

234 

16,913 

Georgia 

255 

7,502 

8 

203 

263 

7,705 

Guam 

317 

431 

4 

5 

321 

436 

Indiana 

298 

8.206 

1 

26 

299 

8,232 

Iowa 

289 

4,025 

1 1 

158 

300 

4,183 

Kentucky 

303 

5,463 

3 

35 

306 

5,498 

Louisiana 

447 

10,132 

6 

1 16 

453 

10,248 

Maine 

103 

819 

0 

0 

103 

819 

Maryland 

269 

8,600 

I 

91 

270 

8,691 

Massachusetts 

318 

8,683 

0 

52 

320 

8,735 

Michigan 

343 

15,387 

4 

161 

347 

15,548 

Minnesota 

278 

7,503 

3 

72 

281 

7,575 

Mississippi 

269 

4,510 

0 

0 

269 

4,510 

Missouri 

454 

9,825 

3 

63 

457 

9,889 

Montana 

177 

733 

2 

8 

179 

741 

Nebraska 

437 

3,361 

1 

8 

438 

3,369 

Nevada 

173 

833 

0 

0 

173 

833 

New  Jersey 

349 

13,858 

1 

44 

350 

13,902 

New  Mexico 

226 

1,987 

19 

213 

245 

2,200 

New  York 

496 

36,060 

7 

495 

503 

36,555 

North  Dakota 

179 

713 

9 

36 

188 

750 

Oregon 

69 

2,778 

0 

0 

69 

2,778 

Pennsylvania 

406 

24,853 

3 

181 

409 

25,034 

Texas 

101 

16,967 

2 

267 

103 

17,235 

Utah 

146 

779 

0 

0 

146 

779 

Vermont 

145 

511 

5 

14 

150 

525 

Wisconsin 

480 

13,567 

0 

0 

480 

13,567 

Wyoming 

85 

228 

2 

6 

87 

234 

Total 

13,714 

303,228 

318 

3,138 

14,032 

306,366 

126 
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Table  7-3 

Unweighted  and  Final  Weighted  Counts  of  Assessed  and  Excluded  Students  by  Jurisdiction 

Grade  8 Public  Schools 


Jurisdiction 

Assessed 

Unweighted  Weighted 

Excluded 

Unweighted  Weighted 

Assessed  and  Excluded 
Unweighted  Weighted 

Alabama 

2,305 

51,710 

210 

4,396 

2,515 

56,106 

Alaska 

1,548 

8,335 

93 

382 

1,641 

8,717 

Arizona 

2,248 

52,529 

205 

4,434 

2,453 

56,964 

Arkansas 

1,879 

32,991 

165 

2,632 

2,044 

35,624 

California 

2,432 

339,680 

303 

37,215 

2,735 

376,895 

Colorado 

2,629 

46,967 

146 

2,330 

2,775 

49,297 

Connecticut 

2,552 

32,922 

274 

3,152 

2,826 

36,075 

Delaware 

1,890 

7,807 

139 

464 

2,029 

8,271 

District  of  Columbia 

1,744 

4,140 

205 

413 

1,949 

4,553 

Florida 

2,498 

144,325 

285 

14,882 

2,783 

159,207 

Georgia 

2,417 

82,372 

200 

6,058 

2,617 

88,430 

Guam 

933 

2,245 

47 

94 

980 

2,340 

Hawaii 

2,266 

12,191 

124 

629 

2,390 

12,821 

Indiana 

2,429 

72,542 

163 

4,458 

2,592 

77,001 

Iowa 

2,277 

35,087 

128 

1,793 

2,405 

36,880 

Kentucky 

2,526 

45,635 

118 

1,959 

2,644 

47,594 

Louisiana 

2,663 

53,687 

190 

3,370 

2,853 

57,056 

Maine 

2,341 

15,778 

149 

912 

2,490 

16,690 

Maryland 

2,198 

53,640 

181 

3,912 

2,379 

57,553 

Massachusetts 

2,405 

57,944 

229 

5,012 

2,634 

62,956 

Michigan 

2,198 

109,908 

132 

5,915 

2,330 

1 15,823 

Minnesota 

2,508 

61,935 

102 

2,391 

2,610 

64,326 

Mississippi 

2,539 

38,421 

202 

2,863 

2,741 

41,284 

Missouri 

2,453 

58,293 

179 

3,857 

2,632 

62,151 

Montana 

1,973 

12,235 

71 

376 

2,044 

12,611 

Nebraska 

2,737 

22,507 

110 

861 

2,847 

23,367 

Nevada 

1,009 

18,821 

85 

1,651 

1,094 

20,472 

New  Hampshire 

1,801 

14,604 

98 

668 

1,899 

15,272 

New  Jersey 

1,691 

77,186 

167 

6,736 

1,858 

83,922 

New  Mexico 

2,505 

21,438 

262 

1,957 

2,767 

23,395 

New  York 

2,025 

179,782 

190 

14,963 

2,215 

194,745 

North  Carolina 

2,706 

85,239 

138 

4,146 

2,844 

89,384 

North  Dakota 

2,688 

9,271 

76 

265 

2,764 

9,536 

Oregon 

2,401 

36,601 

134 

1,748 

2,535 

38,349 

Rhode  Island 

2,184 

9,837 

167 

685 

2,351 

10,522 

South  Carolina 

2,184 

46,657 

166 

3,144 

2,350 

49,801 

Tennessee 

2,391 

60,055 

105 

2,661 

2,496 

62,717 

Texas 

2,382 

245,688 

222 

20,282 

2,604 

265,970 

Utah 

2,766 

36,869 

198 

2,32! 

2,964 

39,190 

Vermont 

2,059 

7,525 

118 

406 

2,177 

7,931 

Virginia 

2,653 

78,426 

208 

5,468 

2,861 

83,894 

Washington 

2,539 

68,561 

153 

3,627 

2,692 

72,188 

West  Virginia 

2,646 

22,706 

250 

1 ,978 

2,896 

24,684 

Wisconsin 

2,216 

63,977 

208 

5,299 

2,424 

69,277 

Wyoming 

2,789 

7,847 

91 

251 

2,880 

8,098 

Total 

102,223 

2,546,920 

7,386 

193,018 

109,609 

2,739,938 

127 
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Table  7-4 

Unweighted  and  Final  Weighted  Counts  of  Assessed  and  Excluded  Students  by  Jurisdiction 

Grade  8 Nonpublic  Schools 


Jurisdiction 

Assessed 

Unweighted  Weighted 

Excluded 

Unweighted  Weighted 

Assessed  and  Excluded 
Unweighted  Weighted 

Alabama 

119 

3,561 

0 

0 

119 

3,561 

Arkansas 

62 

1,467 

0 

0 

62 

1,467 

California 

232 

36,249 

0 

0 

232 

36,249 

Connecticut 

266 

4,761 

2 

32 

268 

4,793 

Delaware 

284 

1,671 

0 

0 

284 

1,671 

District  of  Columbia 

228 

1,022 

0 

0 

228 

1,022 

DoDEA/DDESS 

635 

1,372 

40 

81 

675 

1,453 

DoDEA/DoDDS 

2,215 

4,842 

62 

127 

2,277 

4,968 

Georgia 

267 

5,492 

6 

36 

273 

5,527 

Guam 

203 

560 

0 

0 

203 

560 

Iowa 

284 

4,724 

0 

0 

284 

4,724 

Kentucky 

221 

4,978 

0 

0 

221 

4,978 

Louisiana 

429 

11,073 

6 

151 

435 

11,224 

Maryland 

302 

8,564 

1 

26 

303 

8,590 

Massachusetts 

306 

9,159 

4 

148 

310 

9,307 

Michigan 

293 

13,137 

0 

0 

293 

13,137 

Minnesota 

250 

6,007 

0 

0 

250 

6,007 

Missouri 

355 

7,387 

1 

19 

356 

7,406 

Montana 

121 

578 

4 

19 

125 

597 

Nebraska 

367 

3,332 

1 

14 

368 

3,346 

Nevada 

101 

734 

0 

0 

101 

734 

New  Hampshire 

212 

1,174 

1 

6 

213 

1,180 

New  Jersey 

323 

12,888 

6 

209 

329 

13,097 

New  Mexico 

233 

2,373 

1 

25 

234 

2,398 

New  York 

541 

34,961 

4 

189 

545 

35,150 

North  Dakota 

211 

652 

l 

3 

212 

655 

Oregon 

43 

2,622 

0 

0 

43 

2,622 

Rhode  Island 

425 

1,717 

2 

8 

427 

1,725 

South  Carolina 

164 

3,501 

1 

19 

165 

3,520 

Texas 

168 

17,505 

1 

91 

169 

17,596 

Utah 

40 

750 

1 

16 

41 

766 

Vermont 

115 

380 

0 

0 

115 

380 

Washington 

182 

4,203 

0 

0 

182 

4,203 

Wisconsin 

366 

10,867 

1 

25 

367 

10,892 

Wyoming 

51 

178 

0 

0 

51 

178 

Total 

10,614 

224,438 

146 

1,244 

10/760 

225,682 

12K 


7.4  CHARACTERISTICS  OF  NONRESPONDING  SCHOOLS  AND 
STUDENTS 

In  the  previous  section,  procedures  were  described  for  adjusting  the  survey  weights  so  as 
to  reduce  the  potential  bias  of  nonparticipation  of  sampled  schools  and  students.  To  the  extent 
that  a nonresponding  school  or  student  is  different  from  those  respondents  in  the  same 
nonresponse  adjustment  class,  potential  for  nonresponse  bias  remains. 

In  this  section  we  examine  the  potential  for  remaining  nonresponse  bias  in  two  related 
ways.  First,  we  examine  the  weighted  distributions,  within  each  grade  and  jurisdiction,  of  certain 
characteristics  of  schools  and  mathematics  students,  both  for  the  full  sample  and  for  respondents 
only.  This  analysis  is  of  necessity  limited  to  those  characteristics  that  are  known  for  both 
respondents  and  nonrespondents,  and  hence  cannot  directly  address  the  question  of  nonresponse 
bias.  The  approach  taken  does  reflect  the  reduction  in  bias  obtained  through  the  use  of 
nonresponse  weighting  adjustments.  As  such,  it  is  more  appropriate  than  a simple  comparison  of 
the  characteristics  of  nonrespondents  with  those  of  respondents  for  each  jurisdiction. 

The  second  approach  involves  modeling  the  probability  that  a school  is  a nonrespondent, 
as  a function  of  the  nonresponse  adjustment  class  within  which  the  school  is  located,  together 
with  other  school  characteristics.  This  has  been  achieved  using  linear  logistic  regression  models, 
with  school  response  status  as  the  dependent  variable.  By  testing  to  see  if  the  school 
characteristics  add  any  predictive  ability  to  the  model,  over  and  above  using  the  membership  of 
the  nonresponse  adjustment  class  to  make  this  prediction,  we  can  obtain  some  insight  into  the 
remaining  potential  for  nonresponse  bias.  If  these  factors  are  substantially  marginally  predictive, 
there  is  a danger  that  significant  nonresponse  bias  remains.  These  models  have  been  developed 
for  public  schools  in  each  of  the  12  jurisdictions  in  grade  4 and  1 1 jurisdictions  in  grade  8 having 
public-school  participation  (after  substitution)  of  below  90  percent  (with  a participation  rate  prior 
to  substitution  in  excess  0(70%). 


7.4.1  Weighted  Distributions  of  Schools  Before  and  After  School  Nonresponse 

Tables  7-5  and  7-6  show  the  mean  values  of  certain  school  characteristics  for  public 
schools,  both  before  and  after  nonresponse.  The  means  are  weighted  appropriately  to  reflect 
whether  nonresponse  adjustments  have  been  applied  (i.e.,  to  respondents  only)  or  not  (to  the  full 
set  of  in-scope  schools).  The  variables  for  which  means  are  presented  are  the  percentage  of 
students  in  the  school  who  are  Black,  the  percentage  who  are  Hispanic,  the  median  household 
income  ( 1 989)  of  the  ZIP  code  area  where  the  school  is  located,  and  the  type  of  location.  All 
variables  were  obtained  from  the  sample  frame,  and  so  from  Quality  Education  Data,  Inc.  (QED), 
described  in  Chapter  3,  with  the  exception  of  the  type  of  location.  This  variable  was  derived  for 
each  sampled  school  using  U.S.  Bureau  of  Census  data.  The  type  of  location  variable  has  seven 
possible  levels,  which  are  defined  in  Chapter  3.  Although  this  variable  is  not  interval-scaled,  the 
mean  value  does  give  an  indication  of  the  degree  of  urbanization  of  the  population  represented  by 
the  school  sample  (lower  values  for  type  of  location  indicate  a greater  degree  of  urbanization). 
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Table  7-5 

Weighted  Mean  Values  Derived  from  Sampled  Public  Schools  - Grade  4 


Weighted  Weighted  Mean  Values  Derived  from 

Participation  Weighted  Mean  Values  Derived  Responding  Sample*  with  Substitutes  and 

Rate  After  from  Full  Sample  School  Nonresponse  Adjustment 


Jurisdiction 

Substitution 

(%) 

Percent 

Black 

Percent 

Hispanic 

Median 

Income 

Type  of 
Location 

Percent 

Black 

Percent 

Hispanic 

Median 

Income 

Type  of 
Location 

Alabama 

93.36 

35.26 

0.30 

$23,633 

4.79 

35.69 

0.32 

$23,415 

4.79 

Alaska 

91.12 

3.97 

2.19 

$36,227 

4.87 

4.01 

2 22 

$36,685 

4.87 

Arizona 

87.07 

3.83 

25.17 

$31,014 

3.13 

3.97 

25.53 

$31,099 

3.12 

Arkansas 

77.50 

23.73 

0.59 

$22,309 

5.35 

23.27 

0.45 

$22,203 

5.35 

California 

93.69 

8.76 

36.08 

$34,909 

3.18 

9.02 

36.80 

$34,451 

3.18 

Colorado 

99.06 

4.28 

16.37 

$32,561 

3.77 

4.47 

16.08 

$32,552 

3.77 

Connecticut 

100.00 

13.15 

9.75 

$44,906 

3.74 

13.15 

9.75 

$44,906 

3.74 

Delaware 

100.00 

29.02 

2.99 

$26,200 

4.51 

29.02 

2.99 

$26,200 

4.51 

District  of  Columbia 

100.00 

88.13 

5.54 

$27,908 

1.00 

88.13 

5.54 

$27,908 

1.00 

DoDEA/DDESS 

100.00 

DoDEADoDDS 

100.00 

— 

— 

Florida 

100.00 

24.07 

13.29 

$28,752 

3.26 

24.07 

13.29 

$28,752 

3.26 

Georgia 

98.07 

27.67 

1.20 

$30,602 

4.42 

27.59 

1.20 

$30,767 

4.43 

Guam 

100.00 

2.35 

0.50 

7.00 

2.35 

0.50 

7.00 

Hawaii 

100  00 

3.02 

4.88 

$35,598 

4.02 

3.02 

4.88 

$35,598 

4.02 

Indiana 

90.77 

10.24 

1.99 

$28,289 

4.46 

10.28 

2.00 

$28,461 

4.46 

Iowa 

86.77 

3.35 

1.70 

$27,498 

5.02 

3.54 

1.83 

$27,190 

5.02 

Kentucky 

96.20 

10.39 

0.24 

$23,706 

5.28 

10.22 

0.27 

$23,580 

5.28 

Louisiana 

100.00 

42.07 

1.50 

$23,399 

4.28 

42.07 

1.50 

$23,399 

4.28 

Maine 

86.73 

0.13 

0.54 

$28,824 

5.72 

0.13 

0.53 

$28,885 

5.75 

Maryland 

92.52 

34.46 

2.51 

$40,401 

3.47 

35.48 

2.18 

$39,889 

3.47 

Massachusetts 

97.19 

7.47 

7.90 

$41,309 

4.07 

7.55 

7.88 

$40,973 

4.06 

Michigan 

87.94 

16.23 

2.46 

$33,181 

4.18 

15.29 

2.76 

$33,065 

4.23 

Minnesota 

93.34 

3.81 

1.29 

$33,531 

4.75 

3.98 

1.34 

$33,517 

4.74 

Mississippi 

97.24 

50.99 

0.06 

$20,952 

5.60 

51.08 

0.06 

$20,939 

5.59 

Missouri 

99.49 

15.89 

0.66 

$28,984 

4.61 

15.94 

0.67 

$29,044 

4.61 

Montana 

81.46 

0.52 

1.71 

$24,568 

5.40 

0.56 

1.48 

$24,749 

5.37 

Nebraska 

99.94 

5.12 

2.30 

$28,148 

4.99 

5.12 

2.30 

$28,148 

4.99 

Nevada 

86.11 

9.92 

12.88 

$32,587 

3.80 

10.68 

13.22 

$32,420 

3.77 

New  Jersey 

72.76 

19.75 

11.58 

$43,197 

3.61 

22.03 

10.98 

$42,916 

3.61 

New  Mexico 

100.00 

2.46 

43.96 

$24,326 

4.67 

2.46 

43.96 

$24,326 

4.67 

New  York 

85.71 

17.11 

15.37 

$34,061 

3.31 

16.79 

15.88 

$34,416 

3.31 

North  Carolina 

97.23 

30.66 

0.85 

$28,322 

4.91 

30.31 

0.86 

$28,458 

4.92 

North  Dakota 

96.09 

0.82 

0.79 

$27,007 

5.19 

0.79 

0.73 

$26,990 

5.10 

Oregon 

89.61 

2.57 

4.93 

$29,885 

3.75 

1.42 

5.01 

$29,973 

3.75 

Pennsylvania 

85.63 

12.79 

3.75 

$31,861 

4.27 

12.54 

3.72 

$30,477 

4.28 

Rhode  Island 

99.29 

6.31 

7.49 

$31,867 

3.46 

6.24 

7.47 

$31,699 

3.42 

South  Carolina 

87.85 

40.68 

0.83 

$26,500 

5.11 

41.53 

0.85 

$26,752 

5.11 

Tennessee 

94.20 

23.09 

0.39 

$25,494 

4.23 

22.98 

0.37 

$25,576 

4.23 

Texas 

97.18 

13.52 

33.01 

$28,030 

3.45 

13.78 

32.85 

$28,056 

3.44 

Utah 

100.00 

0.54 

3.68 

$32,427 

4.34 

0.54 

3.68 

$32,427 

4.34 

Vermont 

80.71 

0.57 

0.22 

$31,786 

6.24 

0.60 

0.27 

$31,640 

6.20 

Virginia 

100.00 

21.18 

1.86 

$38,391 

4.19 

21.18 

1.86 

$38,391 

4.19 

Washington 

99.14 

5.05 

5.74 

$34,451 

4 07 

5.07 

5.72 

$34,481 

4.07 

West  Virginia 

100.00 

3.72 

0.25 

$22,247 

5 61 

3.72 

0.25 

$22,247 

5.61 

Wisconsin 

94.26 

10.69 

2.57 

$32,385 

4.47 

10.65 

2.61 

$32,187 

4.47 

Wyoming 

100.00 

1.06 

6 09 

$31,296 

5.43 

1.06 

6.09 

$31,296 

5.43 

Table  7-6 

Weighted  Mean  Values  Derived  from  Sampled  Public  Schools  - Grade  8 


Jurisdiction 


Weighted 
Participation 
Rate  After 
Substitution 

(%) 


Weighted  Mean  Values  Derived 
from  Full  Sample 

Percent  Percent  Median  Type  of 

Black  Hispanic  Income  Location 


Weighted  Mean  Values  Derived  from 
Responding  Sample,  with  Substitutes  and 
School  Nonresponse  Adjustment 
Percent  Percent  Median  Type  of 

Black  Hispanic  Income  Location 


Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District  of  Columbia 
DoDEA/DDESS 


89  64 
9:  2 
87.38 
70.69 
94.06 
100.00 
100.00 
100.00 
100.00 
100.00 


36.17 

4.79 

4.13 

22.65 

7.78 

5.94 

11.71 

27.50 

89.10 


0.20 

2.02 

25.04 

0.34 

33.36 

16.31 

8.77 

3.02 

5.66 


$23,795 

$35,861 

$30,779 

$22,476 

$36,591 

S32.219 

$45,934 

$35,376 

$29,059 


4.91 

4.78 

3.16 

5.44 

3.2f< 

3.70 

3.84 

5.04 

1.00 


36.10 
4.74 
3.82 

22.30 

7.81 

5.94 

11.71 

27.50 

89.10 


0.23 

2.02 

26.08 

0.36 

32.95 

16.31 

8.77 

3.02 

5.66 


$23,603 

$35,982 

$30,376 

$22,577 

$36,473 

$32,219 

$45,934 

$35,376 

$29,059 


4.91 
4.78 
3.17 
5.46 
3.23 
3.70 
3.84 
5.04 
1 .00 


DoDEA/DoDDS 

100.00 





14.76 

$28,865 

Florida 

100.00 

23.64 

14.76 

$28,865 

3.3! 

23.64 

3.31 

Georgia 

99.00 

28.60 

1.14 

$30,466 

4.39 

28.53 

1.14 

$30,568 

4.39 

Guam 

100.00 

3.19 

0 53 

7.00 

3.19 

0.53 



7.00 

hawaii 

99.95 

2.40 

5.45 

$34,998 

4.20 

2.40 

5.45 

$34,999 

4.20 

Indiana 

90.57 

12.07 

2.05 

$28,365 

4.46 

12.32 

2.09 

$28,508 

4.45 

Iowa 

84.09 

3.03 

1.13 

$27,724 

5.01 

3 09 

1.08 

$27,701 

5.02 

Kentuckv 

92.33 

8.64 

0.09 

$23,600 

5.33 

8.85 

0.15 

S23,739 

5.34 

Louisiana 

i 00.00 

42.39 

0.94 

$23,593 

4.56 

42.39 

0.94 

$23,593 

4.56 

Maine 

90.28 

0.15 

0.51 

$29,391 

5.68 

0.16 

0.55 

$29,457 

5 68 

Maryland 

85.59 

32.29 

2.49 

$41,211 

3.60 

32.58 

2.34 

$41,372 

3.60 

Massachusetts 

92.38 

8.05 

7.67 

$41,890 

4.03 

8.22 

7.85 

$42,450 

4.03 

Michigan 

85.71 

16.12 

1 84 

$33,491 

4 29 

17.27 

1.90 

$33,700 

4.30 

Minnesota 

Mississippi 

88.39 

95.36 

3.80 

49.15 

1.31 

0.07 

$32,730 

$20,989 

491 

5.64 

3.77 

48.89 

1.22 

0.07 

$32,849 

$21,020 

4.90 

5.63 

Missouri 

95.96 

14.61 

0.65 

$28,470 

4.81 

14.59 

0.67 

$28,576 

4.82 

Montana 

74.69 

0.28 

1.10 

$25,061 

5.42 

0.34 

111 

$24,843 

5.42 

Nebraska 

99.82 

5.66 

2.72 

$28,687 

4.99 

5.66 

2.73 

$28,683 

4.99 

Nevada 

37  82 

9.77 

13.00 

$32,336 

3.81 

11.15 

14.26 

$31,597 

3 70 

New  Hampshire 

68.77 

0.90 

0.85 

$39,281 

5 27 

0.93 

0.66 

$39,181 

5.33 

New  Jersey 

64.71 

16.90 

11.61 

$43,809 

3.73 

18.66 

13.79 

$42,371 

3.67 

New  Mexico 

100.00 

2.30 

45.45 

$24,205 

4.72 

2.30 

45.45 

$24,205 

4.72 

New  York 

79.58 

19.63 

13.99 

$35,173 

3.32 

18.52 

13.34 

$35,812 

3.36 

North  Carolina 

100.00 

30.67 

0.77 

$28,071 

4.89 

30.67 

0 77 

$28,071 

4.89 

North  Dakota 

94.82 

0.60 

0.65 

$27,692 

5 17 

0.63 

0.61 

$27,519 

5.08 

Oregon 
Rhode  Island 

91.64 

2.19 

4.60 

$29,813 

3 89 

2.41 

4.77 

$29,769 

3.89 

89.94 

6.06 

6 64 

$32,566 

3.53 

6.17 

7 10 

$32,254 

3.52 

South  Carolina 

86.70 

41.49 

0.39 

$26,878 

4.98 

41.60 

0.38 

$27,022 

4.98 

Tennessee 

91.67 

21.13 

0 !9 

$25,640 

4.40 

20  13 

0.21 

$26,070 

4.40 

Texas 

95.11 

12.98 

33.28 

$28,421 

3.51 

13.15 

32.76 

$28,238 

3.48 

Utah 

100.00 

0.60 

3.65 

$32,232 

4.36 

0.60 

3.65 

$32,232 

4.36 

Vermont 

74.18 

0.52 

0.19 

$31,773 

6.12 

0.53 

0 17 

$31,331 

6.06 

Virginia 

Washington 

100.00 

21.10 

1 69 

$38,695 

4.30 

21  10 

1.69 

$38,695 

4.30 

95.13 

4.06 

571 

$34,762 

4 07 

4.07 

5.82 

$34,684 

4.06 

West  Virginia 

100.00 

3.99 

0.06 

$22,565 

5.49 

3.99 

0.06 

$22,565 

5.49 

Wisconsin 

78.30 

7.74 

1.89 

$32,098 

4.63 

8.35 

2.05 

$31,701 

4.64 

Wyoming 

100.00 

0.73 

5 83 

$31,294 

5.41 

0.73 

5.83 

$31,294 

5.41 

Two  sets  of  means  are  presented  for  these  four  variables.  The  first  set  shows  the 
weighted  mean  derived  from  the  full  sample  of  in-scope  schools  selected  for  mathematics;  that  is, 
respondents  and  nonrespondents  (for  which  there  was  no  participating  substitute).  The  weight  for 
each  sampled  school  is  the  product  of  the  school  base  weight  and  the  grade  enrollment.  This 
weight  therefore  represents  the  number  of  students  in  the  state  represented  by  the  selected 
school.  The  second  set  of  means  is  derived  from  responding  schools  only,  after  school 
substitution.  In  this  case  the  weight  for  each  school  is  the  product  of  the  nonresponse-adjusted 
school  weight  and  the  grade  enrollment  of  the  original  school,  and  therefore  indicates  the  number 
of  students  in  the  jurisdiction  represented  by  the  responding  school. 

Tables  7-7  and  7-8  show  some  of  these  same  statistics  for  all  schools  combined,  for  those 
jurisdictions  where  both  the  public-school  participation  rate  prior  to  substitution,  and  the 
nonpublic-school  participation  rate  prior  to  substitution,  exceeded  70  percent.  These  are  the 
jurisdictions  for  which  assessment  results  have  been  published  for  both  public  and  nonpublic 
schools  combined.  Data  on  minority  enrollment  were  not  available  for  nonpublic  schools,  and  so 
are  not  included  in  Tables  7-7  and  7-8. 

The  differences  between  these  sets  of  means  give  an  indication  of  the  potential  for 
nonresponse  bias  that  has  been  introduced  by  nonresponding  schools  for  which  there  was  no 
participating  substitute.  For  example,  in  Kentucky  at  grade  8 the  mean  percentage  Black 
enrollment,  estimated  from  the  original  sample  of  public  schools,  is  8.64  percent  (Table  7-6). 

The  estimate  from  the  responding  schools  is  8.85  percent.  Thus  there  may  be  a slight  bias  in  the 
results  for  Kentucky  because  these  two  means  differ.  Note,  however,  that  throughout  these  four 
tables  the  differences  in  the  two  sets  of  mean  values  are  generally  very  slight,  at  least  in  absolute 
terms,  suggesting  that  it  is  unlikely  that  substantial  bias  has  been  introduced  by  schools  that  did 
not  participate  and  for  which  no  substitute  participated.  Of  course  in  a number  of  states  (as 
indicated)  there  was  no  nonresponse  at  the  school  level,  so  that  these  sets  of  means  are  identical. 
Even  in  those  jurisdictions  where  school  nonresponse  was  relatively  high  (such  as  Arkansas  at 
both  grade  4 and  grade  8 and  Vermont  at  grade  8),  the  absolute  differences  in  means  are  slight. 
Occasionally  the  relative  difference  is  large  (the  “Percent  Black”  in  New  Jersey  at  grade  4,  for 
example),  but  these  are  for  small  population  subgroups,  and  thus  are  very  unlikely  to  have  a large 
impact  on  results  for  the  jurisdiction  as  a whole. 


7.4.2  Characteristics  of  Schools  Related  to  Response 

In  an  effort  to  evaluate  the  possibility  that  substantial  bias  remains  as  a result  of  school 
nonparticipation,  following  the  use  of  nonresponse  adjustments,  a series  of  analyses  were 
conducted  on  the  response  status  for  public  schools.  This  analysis  was  restricted  to  those 
jurisdictions  with  a participation  rate  below  90  percent  (after  substitution),  because  these  are  the 
jurisdictions  where  the  potential  for  nonresponse  bias  is  likely  to  be  the  greatest.  Those 
jurisdictions  with  an  initial  public-school  response  rate  below  70  percent  were  not  included, 
since  NAEP  does  not  report  results  for  these  jurisdictions  because  of  concern  about  nonresponse 
bias.  Information  about  this  can  be  found  in  Chapter  10.  Nonpublic  schools  were  omitted  from 
these  analyses  because  of  the  small  sample  sizes  involved,  which  means  that  it  is  difficult  to 
assess  whether  a potential  for  bias  exists. 


Table  7-7 

Weighted  Mean  Values  Derived  from  All  Sampled  Schools  for 
Jurisdictions  Achieving  Minimal  Required 
Public  and  Nonpublic  School  Participation , Before  Substitution  - Grade  4 


Jurisdiction 

Weighted  Participation 
Rate  After  Substitution  (%) 
Public  Nonpublic 

Weighted  Mean  Values  Derived  from 
Full  Sample 

Median  Income  Type  of  Location 

Weighted  Mean  Values  Derived  from 
Responding  Sample,  with  Substitutes 
and  School  Nonresponse  Adjustment 
Median  Income  Type  of  Location 

Alabama 

71.63 

$23,876 

4.72 

$23,444 

4.71 

Arizona 

87.07 

86.82 

$31,052 

3.14 

$30,907 

3.15 

Arkansas 

77.50 

85.53 

$22,544 

5.22 

$22,470 

5.24 

California 

93.69 

73.45 

$35,445 

3.10 

$35,144 

3.11 

Colorado 

99.06 

76.05 

$32,252 

3.63 

$32,243 

3.62 

Connecticut 

100.00 

74.70 

$44,928 

T.60 

$44,172 

3.60 

Georgia 

98.07 

98.78 

$30,233 

4.30 

$30,387 

4.30 

Guam 

100.00 

77.95 

7.00 

7.00 

Indiana 

90.77 

85.95 

$28,782 

4.38 

$28,991 

4.37 

Iowa 

86.77 

81.98 

$27,472 

4.94 

$27,162 

4.94 

Kentucky 

96.20 

87.03 

$24,425 

5.07 

$24,327 

5.08 

Louisiana 

100.00 

86.05 

$23,643 

4.18 

$23,597 

4.23 

Maine 

86.73 

74.12 

$28,808 

5.69 

$28,797 

5.70 

Massachusetts 

97.19 

84.34 

$40,908 

3.95 

$40,369 

3.94 

Michigan 

87.94 

93.60 

$33,148 

4.06 

$32,821 

4.12 

Minnesota 

93.34 

78.02 

$33,142 

4.74 

$33,007 

4.75 

Mississippi 

97.24 

79.46 

$20,872 

5.48 

$20,861 

5.48 

Missouri 

99.49 

100.00 

$29,883 

4.46 

$29,934 

4.46 

Montana 

81.46 

94.10 

$24,628 

5.34 

$24,817 

5.31 

Nebraska 

99.94 

91.49 

$28,700 

4.82 

$28,756 

4.80 

Nevada 

86.11 

100.00 

$32,572 

3.75 

$32,399 

3.70 

New  Mexico 

100.00 

89.81 

$24,466 

4.65 

$24,302 

4.67 

New  York 

85.71 

91.02 

$33,482 

3.07 

$33,625 

3.09 

Utah 

100.00 

80.93 

$32,541 

4.33 

$32,501 

4.34 

Vermont 

80.71 

74.26 

$31,907 

6.16 

$31,651 

6.08 

Wyoming 

100.00 

95.45 

$31,191 

5.40 

$31,176 

5.40 

» 
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Table  7-8 

Weighted  Mean  Values  Derived  from  All  Sampled  Schools  for 
Jurisdictions  Achieving  Minimal  Required 
Public  and  Nonpublic  School  Participation,  Before  Substitution  - Grade  8 


Jurisdiction 

Weighted  Participation 
Rate  After  Substitution  (9c) 
Public  Nonpublic 

Weighted  Mean  Values  Derived  from 
Weighted  Mean  Values  Derived  from  Responding  Sample,  with  Substitutes 
Full  Sample  and  School  Nonresponse  Adjustment 

Median  Income  Type  of  Location  Median  Income  Tvne  of  I.ocatinn 

California 

94.06 

74.69 

$37,08! 

3.26 

$37,136 

3.23 

Georgia 

99.00 

88.38 

$30,763 

4.21 

$30,931 

4.20 

Guam 

100.00 

76.28 

7.00 

7.00 

Iowa 

84.09 

88.18 

$27,815 

4.89 

$27,731 

4.90 

Louisiana 

100.00 

73.24 

$24,361 

4.28 

$24,239 

4.34 

Massachusetts 

92.38 

73.59 

$41,809 

3.90 

$41,910 

3.85 

Michigan 

85.71 

88.42 

$33,750 

4.21 

$33,743 

4.23 

Minnesota 

88.39 

74.84 

$32,994 

4.77 

533.041 

4.82 

Missouri 

95.90 

l(X).tX) 

$29,636 

4.61 

$29,728 

4.62 

Montana 

74.69 

78.49 

$24,985 

5.40 

$24,754 

5.38 

Nebraska 

99.82 

84.90 

$28,648 

4.81 

$28,597 

4.85 

New  Mexico 

lOO.(K) 

87.48 

$24,083 

4.65 

$23,994 

4.65 

New  York 

79.58 

90.47 

$35,257 

3.15 

$35,623 

3.18 

North  Dakota 

94.82 

85.85 

$27,722 

5.10 

$27,625 

5.(X) 

Rhode  Island 

89.94 

81.32 

S32.318 

3.33 

$31,763 

3.30 

South  Carolina 

86.70 

76.11 

$26,906 

4.90 

$27,025 

4.89 

Texas 

95.1 1 

92.95 

$28,464 

3.46 

$28,319 

3 43 

Vermont 

74.18 

73.14 

$31,729 

6.08 

$31,326 

5.99 

Washington 

95.13 

85.71 

$34,840 

4.(K) 

$34,836 

4.01 

Wsoming 

KX).(X) 

74.49 

$31,232 

5.40 

$31,231 

5.41 

The  12  jurisdictions  investigated  at  grade  4 were  (with  the  public-school  participation 
rate  shown  in  parentheses):  Arizona  (877c),  Arkansas  (787c),  Iowa  (877c),  Maine  (877c), 
Michigan  (887c),  Montana  (817c),  Nevada  (867c).  New  Jersey  (737c),  New  York  (867c), 
Pennsylvania  (86%),  South  Carolina  (88%),  and  Vermont  (8 1 %).  The  jurisdictions  for  grade  8 
were:  Arizona  (87%),  Arkansas  (717c),  Iowa  (847c).  Maryland  (867c),  Michigan  (867c), 
Minnesota  (887c).  Montana  (75%),  New  York  (807c),  South  Carolina  (877c),  Vermont  (747c). 
and  Wisconsin  (78%).  The  approach  used  was  to  develop  a logistic  regression  model  within  each 
jurisdiction  and  grade,  to  predict  the  probability  of  participation  as  a function  of  the  nonresponse 
adjustment  classes  and  other  school  characteristics.  The  aim  was  to  determine  whether  the 
response  rates  are  significantly  related  to  school  characteristics,  after  accounting  for  the  effect  of 
the  nonresponse  class.  Thus  “dummy”  variables  were  created  to  indicate  nonresponse  class 
membership. 

If  there  are  k nonresponse  classes  within  a jurisdiction,  let 

= 1 if  the  school  j is  classified  in  nonresponse  class  / 

= ((otherwise,  for/  = I,. ,„(k-l) 

Within  each  jurisdiction  a logistic  model  was  fitted  to  the  data  on  public-school 
participation.  In  the  model,  the  indicator  variables  for  nonresponse  class  were  included,  and  also 
additional  variables  available  for  participating  and  nonparticipating  schools  alike.  These 
variables  were  the  percentage  of  Black  students  (K,),  the  percentage  of  Hispanic  students  ( Y.A. 
the  estimated  grade  4 or  grade  8 enrollment  size  of  the  school  ( Y.),  and  the  median  household 
income  of  the  ZIP  code  area  in  which  the  school  was  located  ( Yf. 
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The  model  fitted  in  each  jurisdiction  was  the  following: 


Lj  ~ A • t*  + jECjYij . 

Let  P,  denote  the  probability  that  school  j is  a participant,  and  let  Lj  denote  the  logit  of  Pj  That 
is. 


= ln(P,  /(}  - P,)). 


Note  that  this  model  cannot  be  estimated  if  there  are  nonresponse  classes  in  which  all 
schools  participated  (so  that  no  adjustments  for  nonresponse  were  made  for  schools  in  such  a 
class).  Eiven  though  this  analysis  was  restricted  to  those  jurisdictions  with  relatively  poor 
response,  this  occurred  in  a number  of  instances.  When  this  happened,  those  (responding) 
schools  in  such  classes  were  dropped  from  the  analyses.  Tables  7-9  and  7-10  show  the  proportion 
of  the  state  public-school  student  population  that  is  represented  in  the  sample  by  schools  from 
classes  with  less  than  100  percent  response  for  each  grade.  Thus  in  grade  4 for  Maine,  Nevada, 
New  Jersey,  and  Vermont,  there  was  some  nonresponse  within  every  adjustment  class,  whereas 
for  the  other  eight  states  in  grade  4,  some  portion  of  the  population  is  not  represented  because 
schools  were  dropped  from  classes  with  no  nonresponse. 

The  tables  show  that  for  both  grades  only  one  of  the  models  that  contained  all  of  the 
variables  was  significant.  This  was  the  model  for  Arkansas  grade  8 that  has  a p-value  of  .014. 

The  only  individual  variable  for  this  model  that  was  slightly  significant  was  the  dummy  variable 
corresponding  to  nonresponse  class  2,  which  indicates  for  this  state  that  the  nonresponse  classes 
significantly  explain  the  variation  in  the  response  rates.  There  were  two  other  states  in  grade  8, 
Michigan  and  Wisconsin,  that  had  significant  nonresponse  class  variables;  however  in  these  two 
cases,  the  overall  model  was  not  significant. 

For  Pennsylvania  grade  4,  the  single  variable  designating  median  household  income  is 
somewhat  significant.  This  variable  was  used  in  forming  nonresponse  adjustment  classes  in 
Pennsylvania  only  in  large/small  towns  and  rural  areas,  as  minority  enrollment  was  used  in  all 
other  areas  (see  Table  3-4).  From  Table  7-5,  the  median  household  income  for  the  full  sample  is 
S3 1 ,861,  while  for  the  respondents  it  is  $30,477.  This  indicates  that  the  final  sample  is  somewhat 
under-representative  of  schools  with  relatively  high  median  household  income. 

Median  household  income  is  somewhat  significant  for  Michigan  grade  8.  This  variable 
was  used  in  forming  nonresponse  adjustment  classes  in  Michigan  in  large/small  towns  and  rural 
areas.  Minority  enrollment  was  used  in  other  areas  in  Michigan  (see  Appendix  I).  This 
significance  does  not  translate  into  the  results  of  Table  7-6,  since  the  median  household  income 
for  the  full  sample  is  $33,491,  which  is  very  close  to  the  value  of  $33,700  for  the  respondents. 

Another  variable  that  is  significant  for  various  states  is  the  estimated  grade- specific 
enrollment.  This  variable  is  significant  for  Arizona  grade  4 and  Wisconsin  grade  8.  For  public 
schools,  this  variable  was  not  used  in  forming  nonresponse  adjustment  classes  in  these  states  (it 
was  used  only  in  Guam).  This  variable  is  not  shown  in  Tables  7-5  and  7-6.  However,  the  sign  of 
the  coefficient  for  this  variable  in  the  logistic  model  indicates  that  smaller  schools  are  somewhat 
under-represented  in  each  of  these  states. 
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To  determine  if  the  variables  other  than  the  nonresponse  adjustment  class  variables 
added  explanatory'  power  to  the  model,  all  variables  except  the  nonresponse  adjustment  class 
variables  were  tested  collectively  to  see  if  the  estimates  of  the  parameters  were  equal  to  zero. 

This  evaluates  whether,  taken  as  a group,  the  Y variables  are  significantly  related  to  the  response 
probability,  after  accounting  for  nonresponse  class.  The  results  are  shown  in  the  last  columns  of 
Tables  7-9  and  7-10.  None  of  the  tests  were  significant,  which  suggests  that  the  variables  did  not 
add  to  the  model  after  accounting  for  the  nonresponse  adjustment  classes,  even  though  on 
occasion  an  individual  variable  was  significant.  These  results  hold  for  Arkansas  grade  8,  where 
the  full  model  was  significant.  For  Arkansas  grade  8,  the  significant  variable  was  the 
nonresponse  class  2,  which  seems  to  indicate  for  this  state  that  the  nonresponse  adjustment 
classes  alone  explain  the  significant  variations  in  the  probability  of  participation. 

These  results  indicate  that  on  occasion  there  are  differences  between  the  original  samples 
of  schools  and  those  that  participated,  that  are  not  fully  removed  by  the  process  of  creating 
nonresponse  adjustments.  Although  these  effects  are  not  dramatic,  they  are  sometimes 
statistically  significant,  and  may  be  reflected  in  noticeable  differences  in  population 
characteristics  estimated  from  the  respondents,  compared  to  those  obtained  for  the  full  sample. 
However,  the  evidence  presented  here  does  not  permit  valid  speculation  about  the  likely  size  or 
even  direction  of  the  bias  in  mathematics  achievement  results  in  the  few  states  where  these 
sample  differences  are  noticeable. 
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Table  7-9 

Results  of  Logistic  Regression  Analysis  of  School  Nonresponse  - Grade  4 


Table  7-10 

Results  of  Logistic  Regression  Analysis  of  School  Nonresponse  - Grade 


OO 


7.4.3  Weighted  Distributions  of  Students  Before  and  After  Student  Absenteeism 


Tables  7-11  and  7-12  show,  for  the  public  schools  in  each  jurisdiction  and  grade,  the 
weighted  sampled  percentages  of  students  by  gender  (male)  and  race/ethnicity  (White,  not 
Hispanic;  Black,  not  Hispanic;  Hispanic),  SD/LEP  status  for  the  full  sample  of  students  (after 
student  exclusion),  and  for  the  assessed  sample.  The  mean  student  age  in  months  is  also 
presented  on  each  basis.  Tables  7-13  and  7-14  show  these  results  for  both  grades  for  all  students, 
public  and  nonpublic,  in  those  jurisdictions  having  adequate  school  response  rates  to  permit 
reporting  of  combined  results  for  public  and  nonpublic  students. 

The  weight  used  for  the  full  sample  is  the  adjusted  student  base  weight,  defined  in 
Section  7.2.4.  The  weight  for  the  assessed  students  is  the  final  student  weight,  defined  in  Section 
7.3.5.  The  difference  between  the  estimates  of  the  population  subgroups  is  an  estimate  of  the  bias 
in  estimating  the  size  of  the  subgroup,  resulting  from  student  absenteeism. 

Care  must  be  taken  in  interpreting  these  results,  however.  First,  note  that  there  is 
generally  very  little  difference  in  the  proportions  estimated  from  the  full  sample  and  those 
estimated  from  the  assessed  students.  While  this  is  encouraging,  it  does  not  eliminate  the 
possibility  that  bias  exists,  either  within  the  state  as  a whole,  or  for  results  for  gender  and 
race/ethnicity  subgroups,  or  for  other  subgroups.  Second,  on  the  other  hand,  where  differences  do 
exist  they  cannot  be  used  to  indicate  the  likely  magnitude  or  direction  of  the  bias  with  any 
reliability.  For  example,  in  Table  7-12,  for  New  York  the  percentages  of  Black  and  Hispanic 
students  in  the  full  sample  are  respectively  18.33  and  17.26  percent.  For  assessed  students,  these 
percentages  are  16.05  for  Black  students  and  15.84  for  Hispanic  students.  While  these 
differences  raise  the  possibility  that  some  bias  exists,  it  is  not  appropriate  to  speculate  on  the 
magnitude  of  this  bias  by  considering  the  assessment  results  for  Black  and  Hispanic  students,  in 
comparison  to  other  students  in  the  state.  The  reason  is  that  the  underrepresented  Black  and 
Hispanic  students  may  not  be  typical  of  students  that  were  included  in  the  sample,  and  similarly 
those  students  within  the  same  racial/ethnic  groups  who  are  disproportionately  overrepresented 
may  not  be  typical  either.  The  reason  is  that  not  all  students  within  the  same  race/ethnicity  group 
receive  the  same  student  nonresponse  adjustment. 
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Table  7-11 

Weighted  Student  Percentages  Derived  From  Sampled  Public  Schools  - Grade  4 


Weighted 

Student 

Percent 

Jurisdiction 

(%) 

Male 

White 

Black 

Hispanic 

SD 

LEP 

(Months) 

Mate 

Whit* 

Black 

Hispanic 

SD 

LEP 

(Months) 

95.98 

50.21 

58.90 

32.11 

6.12 

4.91 

0.13 

130.49 

50.16 

60.33 

30.83 

6.03 

5.30 

0 12 

130.59 

Alaska 

91.39 

50.09 

58.10 

4.06 

10.57 

9.76 

6.58 

129.95 

49  88 

57.20 

4.03 

11  41 

9.53 

6.76 

129.84 

Ancona 

94.58 

51.08 

55.36 

3.96 

28.66 

3.65 

6.57 

129.49 

51.11 

55.60 

3.91 

29.21 

3.67 

6.38 

129.49 

Arkansas 

96  00 

49.70 

68.46 

20.94 

6.02 

3.14 

0.34 

129.34 

50  00 

68.73 

20.46 

6.21 

3.04 

0.35 

129.39 

94.50 

50.04 

41.72 

8.22 

36.88 

3 73 

14  72 

126.88 

5061 

42.16 

8.31 

36  60 

3.54 

14.96 

126  79 

Colorado 

94.68 

51.35 

68.85 

3 77 

20.52 

6 45 

204 

129.00 

51.27 

68.52 

3.62 

20.76 

6.46 

2.03 

129.06 

Connecticut 

95.67 

50.14 

72.11 

11.12 

12.81 

6.89 

1 21 

127  44 

50.01 

71.93 

11.18 

12.89 

6 89 

1.18 

127.49 

Delaware 

94.40 

49.71 

61.29 

25.53 

9 18 

704 

0.65 

127.12 

4971 

61  51 

24.82 

9.41 

7.04 

0.65 

127  10 

Distric.  of  Columbia 

94.54 

48.82 

6.29 

81.95 

9.64 

1.40 

1.63 

127.77 

48.34 

6.16 

81.56 

10.03 

1.42 

1.61 

127  71 

DuDEA/DDESS 

94.89 

50  88 

48.88 

25  43 

17.76 

4.61 

0 28 

128.21 

50.09 

48.64 

25  25 

18.21 

4.66 

0.22 

128  16 

DoDEA/DoDDS 

94.48 

50  77 

48  00 

18.20 

1568 

4.29 

0.85 

127.83 

50  86 

48.04 

18.52 

I6.(X) 

4 23 

091 

127.80 

93.85 

51  62 

54  92 

20  86 

20  73 

7.70 

3.46 

129.89 

52.03 

54  46 

20.82 

21  10 

7 61 

3 48 

12991 

Georgia 

94.86 

50.17 

57.28 

31.43 

7.49 

5 24 

0.92 

130.49 

50.01 

56  97 

31.26 

7.77 

5.24 

0 93 

130  44 

93.91 

51.80 

7.96 

3 35 

20.29 

1 55 

4.66 

126.72 

52.2! 

7.98 

3 52 

21.66 

1 52 

4.70 

126  68 

95.31 

52  53 

1844 

3.92 

20.39 

5.57 

3 67 

125.81 

52  30 

18.07 

4.07 

21.34 

5 47 

3.74 

125  81 

96.31 

49.05 

8!  63 

9.42 

6 53 

5 56 

0.54 

132  52 

48.82 

82.04 

9.03 

6 47 

5.80 

0.44 

132  46 

96X5 

51  21 

88  16 

2.81 

5.73 

5.76 

()  85 

130  69 

51.21 

88  11 

2.50 

6.11 

5.94 

0.8! 

130  75 

Kentucky 

95.49 

51  66 

84.98 

9 62 

3 7“ 

4 35 

0 05 

129.42 

52  21 

85.04 

9.44 

3.68 

4.22 

0.07 

129.33 

95  34 

50  42 

4891 

40  22 

640 

5 82 

0 67 

131.06 

50.31 

48.98 

39.70 

6 70 

5.78 

0.70 

131.01 

93  53 

49.94 

93  21 

1.02 

3.48 

7.04 

0 22 

130  04 

50  06 

92  88 

1.10 

3 70 

7 01 

0.25 

130.02 

Maryland 

95  70 

49.57 

53.25 

34  26 

7 13 

6 54 

0 39 

125  95 

49  46 

53.36 

33  71 

7.40 

6 52 

0.41 

125.95 

Massachusetts 

94  95 

50.48 

78.63 

6.29 

10  53 

8.12 

1.60 

128.43 

51.2! 

77.17 

6.62 

1!  31 

8 02 

1.80 

128  58 

93  72 

50.23 

73  88 

14  47 

6.75 

4.56 

0 39 

128.66 

5069 

73.71 

1372 

7 65 

4 64 

0 40 

128.66 

94.46 

51  52 

83  58 

4 53 

549 

6 36 

2 13 

130.15 

51.23 

83.32 

4 24 

6.05 

6 42 

2.18 

130.02 

Mississippi 

95  73 

49  59 

44.33 

47  99 

5.19 

3 08 

0.06 

132.37 

50.04 

45  22 

46.88 

5 39 

301 

0 07 

132  42 

95.21 

49  73 

76.69 

13.80 

6.23 

8.44 

0.25 

13134 

49.55 

75  82 

1448 

6.31 

8 27 

0 25 

131  32 

Montana 

96.4 1 

53  82 

81.61 

0.78 

6 55 

531 

0 17 

130.92 

53.36 

79.00 

0.78 

691 

5 40 

0.17 

1 30.88 

94  96 

51.13 

81.95 

5 94 

8 59 

9 76 

1 00 

129.76 

51.32 

81.41 

6 16 

8 7X 

9 72 

1 (X) 

129.70 

92.31 

51  05 

59  8! 

8.19 

22  44 

4.18 

4.27 

128  47 

50.87 

59  95 

7 88 

22  77 

4 39 

4.31 

128.45 

New  Jersey 

94.68 

49  65 

58  85 

20  77 

13.75 

4 04 

1 06 

128  74 

49.39 

58.77 

20.29 

14.02 

3 99 

1 10 

128.73 

New  Mexico 

94.14 

48  59 

42.89 

2 85 

43  34 

7.03 

5 83 

129  91 

48.52 

42  58 

2.83 

43  64 

6 96 

5 91 

1 30.00 

New  York 

93.88 

49.43 

60  30 

16.35 

1727 

4.92 

2.63 

126.99 

49.79 

58.48 

16.16 

1X71 

4 58 

2.65 

126  88 

North  Carolina 

95.62 

50.04 

65  59 

27  00 

4 IX 

6 42 

0 85 

128  80 

50  23 

65  15 

27.10 

4 40 

6 19 

0 XX 

1 28  83 

North  Dakota 

96  20 

50  90 

89.51 

0 80 

4 56 

690 

0 15 

130  57 

50  22 

89  35 

0X2 

4 71 

6.93 

0 19 

1 30  65 

95  13 

49  34 

77  36 

l 72 

11.25 

7 77 

3 45 

128.89 

49.35 

77.33 

1 71 

II  28 

7.66 

3 36 

128  87 

Pennsylvania 

94  77 

50.10 

76  65 

11  29 

9 11 

3 70 

021 

129.64 

50  36 

78  94 

9 57 

8 25 

4.18 

0 15 

129  63 

94  79 

51  45 

75  56 

5 92 

12  98 

9 08 

3 78 

127  54 

52.19 

76  40 

5 42 

12  89 

9 02 

.1  58 

127  19 

South  Carolina 

95  07 

50  20 

53.66 

37  96 

5 73 

6.52 

0 07 

128  68 

50  06 

53  88 

37  40 

5.98 

6 46 

0 07 

12X65 

95  85 

51  10 

72.68 

20  77 

4 15 

7 40 

0 30 

130  19 

51  04 

72  45 

20  X5 

4 37 

741 

0 29 

13013 

95  56 

51  45 

48  78 

13  78 

33  68 

6 05 

1001 

130  54 

51.59 

49  05 

13X9 

33  13 

6 It) 

9 89 

1 30  48 

Utah 

95  27 

5050 

82.22 

l 02 

II  29 

6.87 

1.73 

128  98 

50  27 

81  91 

1.04 

II  67 

691 

1 69 

1 29  07 

95  S2 

50  69 

88  16 

1 42 

6 27 

7 16 

0 28 

128  23 

51  12 

87  91 

1.42 

6 52 

7 45 

0 29 

128  27 

95  13 

49.42 

65  52 

23  92 

571 

5 37 

1 21 

127  98 

49  27 

65  17 

23  84 

5 92 

5 30 

1 29 

128.03 

Washington 

93  80 

51  68 

73  33 

4.65 

10.32 

601 

l 90 

129  53 

51.77 

73  til 

4 67 

HI  62 

5 87 

2 05 

129  53 

West  Viigima 

95  44 

51  48 

87  28 

3 79 

5 84 

5 58 

0 15 

130  16 

51  55 

86  84 

3 90 

6 12 

5 58 

0 16 

130  19 

95  03 

50  54 

80.22 

904 

644 

4 05 

0 90 

1 30.29 

50  52 

80.47 

8 65 

6.54 

4 07 

0 91 

130  33 

Wyoming 

95  69 

*19.92 

81  29 

1 26 

12  35 

X65 

0 66 

1 30  65 

50  43 

80  96 

1 29 

12  75 

8 70 

O 67 

lit)  57 
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Table  7-12 

Weighted  Student  Percentages  Derived  from  Sampled  Public  Schools  - Grade  8 


Weighted  Weighted  Estimates  Derived  from  Assessed  Sample, 

Student  Weighted  Estimates  Derived  form  Full  Sample  with  Student  Nonresponse  Adjustment 

Participation  Percent  Percent  Percent  Percent  Percent  Percent  Mean  Age  Percent  Percent  Percent  Percent  Percent  Percent  Mean  Age 


Jurisdiction 

—(»)  - 

Male 

White 

Black 

Hispanic 

SD 

LEP 

(Months) 

Male 

White 

Black 

Hispanic 

SD 

LEP 

(Months) 

Alabama 

92.74 

48  77 

59.13 

34  28 

3.97 

5.77 

0.04 

179  91 

48  92 

58  99 

34  08 

431 

5 72 

0 10 

179  79 

Alaska 

80.48 

51.69 

67.48 

4 48 

5 54 

7 67 

3 95 

177  21 

51  49 

68  18 

4.04 

6 57 

7.65 

3.93 

177  06 

Arizona 

90.50 

49  44 

57  87 

3 13 

30  45 

4 38 

561 

178  14 

48  51 

57  94 

3 07 

30  44 

4.38 

5.58 

178.14 

Arkansas 

92  35 

49  84 

74.31 

2009 

2.96 

3.82 

0 42 

179  31 

50.02 

74  04 

20  02 

3.10 

3.X8 

047 

179.24 

California 

90  47 

49.09 

37.81 

9 22 

39  84 

390 

7 76 

175  76 

48  89 

39  .39 

7 82 

38  54 

.3.94 

7 55 

175  65 

Colorado 

90.97 

51.29 

69.82 

5.09 

20.80 

6.90 

1 13 

177  99 

51  01 

69.75 

5 03 

20  70 

6 88 

1 15 

177  95 

Connecticut 

90.67 

50.74 

75.93 

9 90 

10  68 

6 24 

0 53 

176  21 

50  73 

7641 

937 

10  73 

6.27 

0.47 

1 76  20 

Delaware 

90  26 

49  54 

65.71 

24  M 

5.72 

7 02 

0 34 

176  46 

49.58 

65  82 

23  47 

5 90 

7 02 

0 36 

176.39 

District  of  Columbia 

84.70 

46.60 

3.38 

83  65 

10  22 

2 59 

2 02 

178  46 

46  47 

3 59 

82  22 

II  23 

2 56 

2.05 

178  09 

DoDEA/DDESS 

95  24 

50  82 

40.62 

30  42 

21  98 

6 03 

0 46 

176  64 

51.52 

40  71 

29  71 

22  24 

6 03 

0 46 

176.63 

DoDE  A/DoDDS 

93  56 

51  99 

45.75 

20  32 

15  28 

4 03 

0 50 

176  02 

51  76 

45  69 

20  32 

15.58 

3 99 

0.55 

176.04 

Florida 

90  72 

48  22 

55  33 

21  48 

20.40 

6 02 

I 45 

179  22 

47  79 

54  12 

21  7! 

21  17 

5 82 

1 65 

179  06 

Georgia 

90  20 

49.93 

57  90 

35.35 

4.22 

3 53 

0 40 

179  32 

49  63 

57  24 

.35  6! 

4 54 

3.55 

038 

179.19 

Guam 

86  21 

53  34 

3 76 

1.26 

14  77 

2 62 

0 78 

175.97 

52  39 

4 05 

1.22 

17  12 

2 68 

0 72 

175  80 

Hawaii 

90.65 

51.71 

15.89 

2 56 

15  68 

534 

2 24 

174.92 

51  76 

15  59 

2 59 

17  16 

5 14 

2.40 

174.75 

Indiana 

92.64 

51.38 

82  26 

10.71 

5 33 

6 76 

0 3X 

179.21 

50  79 

82  37 

10  .39 

5.52 

6 72 

0.37 

179  30 

Iowa 

93  39 

52.1 1 

90  77 

3.00 

3 14 

7 94 

0.34 

178  87 

51  78 

91  20 

2.78 

.3  08 

8 01 

0.30 

178  72 

Kentucky 

93  92 

51.64 

87  04 

9 20 

2 23 

5 22 

0 IX 

179  13 

5I.4X 

86  65 

9 26 

2 42 

5.27 

0 19 

179  08 

Louisiana 

89  16 

48.23 

53  53 

40  46 

3 82 

4 22 

0.35 

179  95 

48  43 

52  93 

40.74 

406 

4 16 

0.41 

179.71 

Maine 

91.58 

49.72 

94  92 

0 49 

1 92 

6 04 

0 44 

179  09 

50  01 

94  72 

0 50 

20! 

6.06 

0 43 

179  13 

Maryland 

91.27 

49,99 

55  43 

33  47 

4 90 

5 50 

0 21 

174.98 

49.64 

55  50 

32  74 

5 25 

5 52 

0 20 

174  95 

Massachusetts 

92  34 

51  75 

80  03 

6 77 

7 92 

8 7.3 

0 94 

177  37 

51  72 

80  21 

6 52 

7 84 

8 7? 

0 90 

177  29 

Michigan 

90  33 

49.66 

73  59 

1748 

4.73 

3 49 

0 67 

178  12 

49  82 

75  58 

1521 

5 1 1 

3 48 

0 62 

178  10 

Minnesota 

91  XI 

50.83 

85.53 

4 56 

3 15 

6 87 

071 

178  08 

50  57 

85  84 

3 98 

3 2(1 

6 91 

0.7.3 

178  01 

Mississippi 

92.56 

48.09 

48  54 

44  87 

4 92 

4 54 

0 04 

1 81  89 

47  95 

48  It 

44  80 

5 33 

441 

0 04 

181  72 

Missouri 

91  41 

49  80 

81  01 

13.51 

3 II 

5 60 

0 15 

179  33 

49.34 

82  03 

12  17 

3 32 

5 68 

0 13 

179  37 

Montana 

92  32 

50  41 

83  17 

0 38 

4 51 

6 14 

0 20 

179.44 

50  12 

83  46 

0.39 

4 58 

6 31 

021 

179.26 

Nebraska 

91  01 

50.60 

86.82 

4 52 

5.69 

8 06 

0 43 

178  06 

50  77 

86  59 

4 62 

5 78 

8.19 

051 

178.09 

Nevada 

90  3! 

50.03 

60  22 

6 40 

25  81 

4 88 

3 ‘A) 

176  94 

50  13 

60  08 

6 19 

25  75 

4 70 

4 02 

176  80 

New  Hampshire 

89.44 

50.22 

92.12 

(1  70 

.3  in 

10  68 

0 12 

179  20 

49  7.3 

91  58 

0.55 

3 48 

10  68 

0 1? 

179  18 

New  Jersey 

92  71 

48  66 

59.05 

16  31 

17  70 

4 48 

0 89 

176  69 

48  50 

59  25 

15  60 

17  97 

4.36 

0 93 

176  64 

New  Mexico 

9000 

48  19 

35  63 

2 32 

51  20 

8 72 

2 09 

178  14 

47  65 

.35  79 

2 33 

51  18 

8 54 

2 26 

178  13 

New  York 

91)76 

51  12 

57  05 

1833 

17  26 

5 26 

1 66 

176  02 

50  48 

60  19 

16  05 

15  84 

5.09 

l 75 

176  03 

North  Carolina 

91  01 

48  30 

64  05 

28  06 

3 50 

4 69 

0 35 

177  80 

48  05 

64  18 

27  48 

.3  80 

4 66 

0.37 

177  78 

North  Dakota 

94  30 

50  63 

91  37 

1 1 1 

2 69 

7 11 

0 44 

178  5? 

50  72 

92  28 

0 9| 

2 58 

6 86 

0 28 

178  57 

Oregon 

90.06 

50  30 

82  24 

2 78 

7 25 

651 

0 92 

177  75 

50  45 

82  45 

2 62 

7 44 

6 58 

0 89 

177  59 

Rhode  Island 

89  50 

48.36 

79  57 

5 07 

9 91 

K7X 

2 30 

176  91 

48  71 

79  06 

.5  04 

10  44 

8 63 

2 43 

1 76  67 

South  Carolina 

89  13 

47.68 

53  02 

40  44 

3 83 

3 98 

0 13 

178  5.3 

47  10 

52  92 

.39  88 

4 15 

3 98 

0 1< 

178  45 

Tennessee 

90  51 

50  30 

78  20 

17  77 

2 75 

7 23 

0 06 

179  62 

Ml  35 

77  80 

17  79 

3 01 

7 23 

0 06 

17952 

Texas 

92.30 

47  82 

47  14 

12  19 

37  25 

5 91 

4 79 

179  49 

47.68 

47.78 

1 1 47 

37  05 

5 75 

4 82 

179  42 

Utah 

90  62 

5009 

87  63 

0 56 

7 53 

5 22 

0 17 

176  96 

M)  .39 

87  23 

0 56 

7 79 

5.20 

0 20 

176  98 

Vermont 

93.21 

51  15 

93  19 

0 68 

2 82 

7 16 

04! 

177  27 

50  89 

93  04 

0 69 

2 92 

7 12 

045 

177  24 

Virginia 

91  20 

49  57 

65  64 

24  44 

4 75 

591 

1 59 

177  37 

49  72 

65  66 

24  12 

5 00 

5 74 

1 (4 

177.33 

Washington 

89  96 

5095 

76  59 

4 IK 

9 16 

6X4 

1 23 

178  58 

50  99 

76  21 

4 12 

9 51 

6 64 

1 42 

178.48 

West  Virginia 

92  44 

50  09 

92  32 

3 05 

2 47 

5 08 

003 

179  04 

50  30 

91  9.3 

3 07 

2 69 

5 08 

0 04 

178  91 

Wisconsin 

92  16 

51  05 

84  05 

7 14 

4 88 

4 46 

0 33 

178  39 

50  84 

X4  29 

6 60 

5 1 1 

4 45 

0 35 

1 78  29 

Wyoming 

92  56 

50  45 

86  61 

0 75 

8 90 

6 94 

0 U 

178 

50  8 3 

86  |6 

0 69 

')  24 

6 90 

0 40 

178  S6 
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Table  7-13 

Weighted  Student  Percentages  Derived  From  All  Schools  Sampled  - Grade.  4 


Jurisdiction 

Weighted 

Student  Participation 
(%) 

Public  Noumiblic 

Percent 

Male 

Weighted  Estimates  Derived  front  Assessed  Sample, 

Weighted  Estimates  Derived  from  Full  Sample  with  Student  Nonresponse  Adjustment 

Percent  Percent  Percent  Percent  Percent  Mean  Age  Percent  Percent  Percent  Percent  Percent  Percent  Mean  Age 
White  Black  Hispanic  SD  LEP  (Months)  Mole  White  Black  Hispanic  SD  LF.P  (Months) 

Alabama 

96.74 

zmm 

61.69 

29.84 

5 59 

4 32 

0 12 

130.33 

50  04 

62.54 

29  OX 

5.60 

4.76 

0.1  1 

130.45 

Arizona 

94.58 

98.89 

50.81 

54.84 

4.02 

27.65 

3 64 

9.15 

129  40 

51  (XI 

55  2X 

4 00 

27  97 

3 62 

8.91 

129  41 

Arkansas 

96.00 

96.69 

49.73 

69.96 

19  61 

5 87 

3.(8) 

0 32 

129  36 

50.03 

70  ]7 

19  20 

6 07 

3 (X) 

0 32 

1 29  39 

California 

94.50 

97.74 

50.17 

42.83 

7.36 

36.37 

3 23 

12  76 

12701 

50  68 

43  OX 

7 56 

36  2 1 

3 n 

13  23 

126.9| 

Colorado 

94.68 

95.83 

51.12 

68  54 

4.84 

20.02 

6 15 

I 'X) 

128.97 

51  12 

6X  17 

4 75 

20  27 

6 16 

1 XX 

1290! 

Connecticut 

95.67 

95  52 

50.59 

72  69 

10  46 

12.85 

6 80 

! 16 

127.36 

50  4X 

72  54 

10  46 

12  99 

6 XO 

I 13 

12741 

Georgia 

94  86 

93.55 

50.15 

58  52 

30  25 

7.25 

5 01 

0 85 

PO  44 

5001 

5X  17 

30.12 

7 52 

501 

0 X5 

130  40 

Guam 

93.91 

94.36 

50.98 

9 55 

3 16 

19  02 

1 M 

4 tX) 

126  54 

51  32 

9 65 

.3.32 

20  29 

1 29 

403 

126  52 

Indiana 

96.3! 

96.44 

49.65 

82  40 

8 83 

6 50 

5.11 

0.4X 

1 32  3X 

49  52 

X2  S') 

X.45 

6 46 

5 '() 

0 39 

132.33 

Iowa 

96.85 

95  87 

51  15 

88.67 

2 51 

5 7 1 

5.31 

0 75 

1 30  66 

51  12 

XS  55 

2 26 

6 07 

5 51 

0 72 

130  73 

Kentucky 

95.49 

96.70 

51  95 

84  92 

9.53 

3 68 

3.93 

0 04 

1 29  49 

52.54 

85  02 

9 33 

3 62 

3.X5 

0 07 

129.40 

Louisiana 

95.34 

96.56 

50.12 

54.49 

35  47 

6.07 

5 47 

0 57 

130  67 

49  09 

54.47 

35  Oh 

6 35 

5 44 

0 60 

1 30  62 

Maine 

93.53 

96.52 

49.88 

93  25 

105 

3 53 

6 87 

0 21 

130  04 

50  12 

92.94 

! 12 

3.73 

6X3 

o 24 

13001 

Massachusetts 

94.95 

95.78 

50.53 

79  05 

6 2X 

1041 

8 51 

1 42 

12X  62 

5 1 06 

77  68 

6 61 

II  15 

X 39 

1.5X 

128.7? 

Michigan 

93.72 

96.73 

50.56 

73  72 

14  33 

7 20 

4 17 

o 3x 

1 28  65 

50  97 

73  41 

13  90 

7 92 

4.26 

0.39 

12X.67 

Minnesota 

94.46 

95  54 

50  77 

84  33 

4 34 

5 40 

5 95 

1 XX 

130  2X 

50  47 

84  .OX 

4.05 

5 92 

6 01 

1 94 

13017 

Mississippi 

95.73 

95.96 

49.91 

4X.9X 

42  7? 

4 86 

3.22 

(106 

132  31 

50  36 

49  50 

42  02 

5 07 

3 15 

0 06 

132  35 

Missouri 

95  21 

95  34 

49  23 

77  67 

12  68 

6 55 

7 57 

((22 

131  31 

49  01 

76  93 

13.28 

6 59 

7 45 

o 22 

131  29 

Montana 

96.41 

95.02 

53  69 

X2.26 

0 73 

6 61 

5 07 

(l  16 

1 30  97 

53  36 

79  67 

073 

6 96 

5 19 

0 16 

130  93 

Nebraska 

94.96 

9X  74 

50.01 

82  85 

5 34 

X 33 

X.57 

091 

129  78 

50.25 

82.46 

5.4X 

x 51 

X 61 

0 92 

129  75 

Nevada 

92  31 

96  46 

51.20 

60  31 

8.23 

2 1 96 

4 (X) 

4 OX 

12X49 

50,95 

6041 

7 94 

22  31 

4 21 

4 14 

12X47 

New  Mexico 

94  14 

94.21 

48.68 

41  23 

2 73 

42  14 

6 46 

6 51 

129  7X 

4X64 

41  02 

2 72 

42  y* 

6 40 

6 5X 

1 29  X6 

New  York 

93.88 

95  92 

50.51 

60.33 

P 66 

16  22 

4 21 

3 4X 

126  99 

50  7X 

59  »5 

1 ‘'O’’ 

P 36 

3 94 

.3  33 

126X9 

Utah 

95  27 

95.35 

50  65 

82  22 

i 02 

u :7 

6 71 

I 69 

12X96 

50  42 

XI  89 

1 05 

1 1 .65 

t>  /> 

1 65 

129  04 

Vermont 

95.52 

97  13 

50.60 

88  33 

I 41 

6 IX 

6 70 

0 27 

12X22 

50  97 

KX.OX 

1 12 

6 43 

6 9X 

o 2 7 

12X  25 

Wyoming 

95.69 

96.26 

50  16 

81  42 

1 23 

12.2“ 

X 46 

0 64 

l V)  69 

50  60 

81  It 

1 25 

!2  67 

8 46 

0 65 

1.3060 

Table  7-14 

Weighted  Student  Percentages  Derived  From  All  Schools  Sampled  - Grade  X 

Weighted 

Student  Participation 
<*) 

jurisdiction  Public  Nonmiblic 

Weighted  Estimates  Derived  from  Full  Sample 
Percent  Percent  Percent  Percent  Percent  Percent  Mean  Age 
Male  Whits  Black  Hispanic  SD  LHP  (Months) 

Weighted  Estimates  Derived  from  Assessed  Numple. 
with  Student  Nonresponse  Adjustment 
Percent  Percent  Percent  Percent  Percent  Percent  Mean  \ge 
Male  White  Block  Hispanic  Sl>  l.EP  (Months) 

California 

‘81.47 

97  AM 

48  63 

39  84 

8 94 

37  56 

3 52 

6 92 

175  ox 

4K  58 

41  10 

7 69 

36  57 

.1  Ml 

6 82 

l 75  59 

Georgia 

90  20 

96  98 

49  48 

59  65 

3 3 60 

4 26 

3 3f, 

0 37 

179  21 

49.26 

59  04 

33  S3 

4 5(, 

3 38 

0 36 

179  08 

Guam 

86  2 1 

94  XX 

53  30 

4 92 

1 29 

13  23 

2 10 

oxi 

175  46 

52  33 

5(14 

1 IK 

15  IX 

2 1 5 

0 70 

P5  30 

lowa 

93  39 

95  97 

51  45 

91  10 

2 7| 

3 74 

7 08 

O 3(1 

1"8  '6 

51  23 

91  42 

2 51 

t 2“ 

7 ' l 

0 27 

1 78  63 

Louisiana 

89  16 

95  53 

48  78 

58  55 

34  72 

4 10 

.3  83 

0 29 

P9  19 

4S  50 

57  96 

34  97 

4 15 

3 -X 

O 31 

PX99 

Massachusetts 

92  74 

95  43 

52  23 

KO  71 

6 29 

7 7| 

K 52 

0X1 

177  22 

52  5S 

81  14 

6 0S 

” 44 

8 1 1 

0 -K 

177  | 3 

Michigan 

90.33 

95  68 

49,56 

73.35 

P 15 

4 T 

< 20 

0 59 

P‘92 

49  M 

7S  25 

15  13 

5 OX 

3 21 

0 55 

P7  91 

Minnesota 

91  XI 

95  58 

51  13 

86.13 

4 20 

1 10 

6 ;x 

0 70 

PX  1“ 

50  97 

86  <8 

3 68 

3 1 5 

O 3 3 

0 72 

PK  i 3 

Missouri 

91,41 

96  29 

49  76 

81  X9 

12  57 

3 02 

5 IX 

0 14 

P9  IS 

49  3 1 

*2  92 

11  2 1 

3 20 

5 24 

0 1 2 

1 79  2 1 

Montana 

92  32 

95  17 

50  01 

K7(X> 

0 76 

4 55 

5 uo 

o 22 

P9  4 < 

49  75 

S3  3(1 

0 37 

4 61 

0 1 3 

0 24 

1 ”9  26 

Nebraska 

91  OJ 

95  48 

50  64 

87  44 

4 10 

5 74 

7 M 

0 19 

1-803 

50  8 3 

XT  21 

4 !X 

5 85 

7 45 

0 40 

PX  05 

New  Mexico 

90(81 

88  68 

47  99 

35  0! 

2 13 

48  71 

X 29 

! 87 

PX  1! 

J-  4X 

V5  60 

2 16 

49  25 

X 1 5 

2 03 

PX  10 

New  York 

90  76 

94  82 

50  16 

55  73 

19  6! 

IP’ 

4 74 

1 79 

175  ?2 

49  9 3 

58  47 

1 7 47 

16  58 

4 29 

1 49 

175-3 

North  Dakota 

94.3(1 

95  99 

50  64 

91  3(1 

1 07 

2 59 

6 75 

11  9 1 

r*  5 1 

50  84 

92  1 5 

0 85 

2 SO 

(,  56 

0 7(| 

178  54 

Rhode  Island 

K9  50 

96  37 

48  48 

79  47 

5 71 

9 M 

: 6i 

1 9r. 

ro  r >6 

48  8| 

?X  96 

5 71 

1007 

7 48 

2 07 

176  47 

South  Carolina 

X9.I3 

95.84 

47.78 

55  80 

37  90 

3 Mi 

3 69 

0 12 

1 78  35 

47  26 

55  70 

3?  37 

3 <81 

3 70 

0 1 1 

PX  27 

Texas 

92.30 

92  50 

47  60 

47  68 

1 1 47 

30  99 

5 49 

4 52 

179  39 

47  64 

4X  30 

1085 

30  70 

5 37 

4 57 

179  .35 

Vermont 

93  21 

94  60 

51  32 

9.3  24 

0 6X 

2 K9 

< 92 

0 39 

P7  20 

51  16 

93  09 

0 69 

3 00 

O 88 

0 4 3 

P7  1 6 

Washington 

89.96 

96  52 

50  81 

76  99 

4 |() 

X 94 

6 44 

1 15 

PK44 

50  84 

76  (41 

100 

9 29 

(■  26 

1 3.1 

1 *8  s' 

Wyoming 

92.56 

97  OK 

50.28 

X6  72 

0 77 

8 85 

o xr 

II  3 3 

PK  5x 

50  OX 

X6  2X 

0 6~ 

9 18 

O x: 

0 3*) 

PX  86 

best  COPY  AVWL^blE 


One  other  feature  to  note  is  that,  for  assessed  students,  information  as  to  the  student’s 
gender  and  race/ethnicity  is  provided  by  the  student,  while  for  absent  students  this  information  is 
provided  by  the  school.  Evidence  from  past  NAEP  assessments  (see,  for  example.  Rust  & 
Johnson,  1992)  indicates  that  there  can  be  substantial  discrepancies  between  those  two  sources, 
especially  with  regard  to  classifying  grade  4 students  as  Hispanic. 

7.5  VARIATION  IN  WEIGHTS 

After  computation  of  full-sample  weights,  an  analysis  was  conducted  on  the  distribution 
of  the  final  student  weights  in  each  jurisdiction.  The  analysis  was  intended  to  ( I)  check  that  the 
various  weight  components  had  been  derived  properly  in  each  jurisdiction,  and  (2)  examine  the 
impact  of  the  variability  of  the  sample  weights  on  the  precision  of  the  sample  estimates,  both  for 
the  jurisdiction  as  a whole  and  for  major  subgroups  within  the  jurisdiction. 

The  analysis  was  conducted  by  looking  at  the  distribution  of  the  final  student  weights  for 
the  assessed  students  in  each  jurisdiction  and  grade  separately  by  public  and  nonpublic  schools. 
Two  key  aspects  of  the  distribution  were  considered  in  each  case:  the  coefficient  of  variation 
(equivalently,  the  relative  variance)  of  the  weight  distribution;  and  the  presence  of  outliers-that 
is,  cases  whose  weights  were  several  standard  deviations  away  from  the  median  weight. 

It  was  important  to  examine  the  coefficient  of  variation  of  the  weights  because  a large 
coefficient  of  variation  reduces  the  effective  size  of  the  sample.  Assuming  that  the  variables  of 
interest  for  individual  students  are  uncorrelated  with  the  weights  of  the  students,  the  sampling 

r -i? 

C " 

variance  of  an  estimated  average  or  aggregate  is  approximately  ( / + J times  as  great  as  the 

corresponding  sampling  variance  based  on  a self-weighting  sample  of  the  same  size,  where  C is  the 
coefficient  of  variation  of  the  weights  expressed  as  a percent.  Outliers,  or  cases  with  extreme 
weights,  were  examined  because  the  presence  of  such  an  outlier  was  an  indication  of  the  possibility 
that  an  error  was  made  in  the  weighting  procedure,  and  because  it  was  likely  that  a few  extreme 
cases  would  contribute  substantially  to  the  size  of  the  coefficient  of  variation. 


In  most  jurisdictions,  the  coefficients  of  variation  were  35  percent  or  less,  both  for  the 

r c f 

whole  sample  and  for  all  subgroups.  This  means  that  the  quantity  (7+  ) was  generally 

below  1.1,  and  the  variation  in  sampling  weights  had  little  impact  on  the  precision  of  sample 
estimates. 


A few  relatively  large  student  weights  were  observed  in  each  grade  in  some  jurisdictions. 
An  evaluation  was  made  of  the  impact  of  trimming  these  largest  weights  back  to  a level 
consistent  with  the  largest  remaining  weights  found  in  the  state  and  grade.  Such  a procedure 
produced  an  appreciable  reduction  in  the  size  of  the  coefficient  of  variation  for  these  weights, 
and  hence  this  trimming  was  implemented.  Westat  judged  that  this  procedure  had  minimal 
potential  to  introduce  bias,  while  the  reduction  in  the  coefficient  of  variation  of  the  weights  gives 
rise  to  an  appreciable  decrease  in  sampling  error  for  the  jurisdictions  and  grades. 
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7.6  CALCULATION  OF  REPLICATE  WEIGHTS 


A replication  method  known  as  jackknife  was  used  to  estimate  the  variance  of  statistics 
derived  from  the  full  sample.  The  process  of  replication  involves  repeatedly  selecting  portions  of 
the  sample  (replicates)  and  calculating  the  desired  statistic  (replicate  estimates).  The  variability 
among  the  calculated  replicate  estimates  is  then  used  to  obtain  the  variance  of  the  full-sample 
estimate. 

In  each  jurisdiction,  replicates  were  formed  in  two  steps.  First,  each  school  was  assigned 
to  one  of  a maximum  of  62  replicate  groups,  each  group  containing  at  least  one  school.  In  the 
next  step,  a random  subset  of  schools  (or,  in  some  cases,  students  within  schools)  in  each 
replicate  group  was  excluded.  The  remaining  subset  and  all  schools  in  the  other  replicate  groups 
then  constituted  one  of  the  62  replicates.  The  process  of  forming  these  replicate  groups,  core  to 
the  process  of  variance  estimation,  is  described  below. 


7.6. 1 Defining  Replicate  Groups  and  Forming  Replicates  for  Variance  Estimation 

Replicate  groups  were  formed  separately  for  public  and  nonpublic  schools.  Once 
replicate  groups  were  formed  for  all  schools,  students  were  then  assigned  to  their  respective 
school  replicate  groups. 

Public  Schools.  Noncertainty  schools  were  sorted  by  jurisdiction  according  to  sample 
type.  Then  within  sample  type,  the  schools  were  sorted  by  new  school  status  and  the  order  in 
which  they  were  selected  from  the  sampling  frame.  The  schools  were  then  grouped  in  pairs. 
Where  there  was  an  odd  number  of  schools,  the  last  replicate  group  contained  three  schools 
instead  of  two.  In  those  jurisdictions  where  the  number  of  pairs  exceeded  62  (Nebraska,  grade  4), 
the  pair  numbering  proceeded  up  to  62,  and  then  decreased  back  from  62  for  the  last  few  pairs. 

Each  of  the  certainty  public  schools  was  assigned  to  one  or  more  replicate  groups  of  its 
own.  If  a school  was  selected  three  or  more  times  in  the  sampling  process,  then  it  was  assigned  to 
two  replicate  groups.  Here,  schools  were  sorted  by  the  estimated  grade  enrollment  prior  to  group 
assignments.  Again,  depending  on  the  jurisdiction,  a maximum  of  62  certainty  groups  was 
formed.  The  group  numbering  resumed  from  the  last  group  number  used  for  the  noncertainty 
schools  if  the  total  number  of  public-school  groups  was  less  than  62.  Otherwise,  the  numbering 
started  from  62  down  to  the  number  needed  for  the  last  certainty  public  school.  In  the  District  of 
Columbia  grade  4,  which  had  only  109  certainty  schools  (no  noncertainty  schools),  groups 
started  at  1 and  continued  up  to  62  and  then  back  down  to  16.  In  the  District  of  Columbia  grade 
8,  which  had  only  36  certainty  schools,  the  groups  went  from  I to  53.  Eighteen  of  the  36 
certainty  schools  in  the  District  of  Columbia  were  selected  three  or  more  times  and  thus  were 
assigned  to  two  replicate  groups.  A replicate  was  formed  by  randomly  deleting  one  half  of  the 
students  in  a certainty  school  from  the  sample.  For  certainty  schools  that  were  assigned  to  two 
replicate  groups,  the  students  were  split  equally  between  four  “halves,”  two  halves  in  each  of  the 
two  replicate  groups.  This  was  repeated  for  each  certainty  school. 

The  purpose  of  this  scheme  was  to  assign  as  many  replicates  to  a jurisdiction’s  public 
schools  as  permitted  by  the  design,  to  a maximum  of  62.  When  more  than  62  replicates  were 
assigned,  the  procedure  ensured  that  no  subset  of  the  replicate  groups  (pairs  of  noncertainty 
schools,  individual  certainty  schools,  or  groups  of  these)  was  substantially  larger  than  the  other 
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replicate  groups.  The  aim  was  to  maximize  the  degrees  of  freedom  available  for  estimating 
variances  for  public-school  data. 

A single  replicate  estimate  was  Turned  by  dropping  one  member  assigned  to  a particular 
replicate  group.  This  process  was  repeated  successively  across  replicate  groups,  giving  up  to  62 
replicate  estimates. 

Nonpublic  Schools.  Replicate  groups  for  noncertainty  nonpublic  schools  were  formed  in 
one  of  the  two  methods  described  below.  If  any  of  the  following  conditions  was  true  for  a given 
jurisdiction,  then  the  subsequent  steps  were  taken  to  form  replicate  groups.  Here,  the  numbering 
started  at  62  down  to  the  last  needed  number. 

Conditions  for  Method  I: 

• fewer  than  1 1 nonpublic  noncertainty  schools;  or 

• fewer  than  2 Catholic  noncerlainty  schools;  or 

• fewer  than  2 non-Catholic  noncertainty  schools. 

Steps  for  Method  I: 

• all  schools  were  grouped  into  a single  replicate  group; 

• schools  were  randomly  sorted;  and 

• starting  with  the  second  school,  replicates  were  formed  by  consecutively 
leaving  out  one  of  the  remaining  n - J schools;  each  replicate  included  the 
first  school. 

When  a given  jurisdiction  did  not  match  conditions  of  the  first  method  (i.e.,  when  all  of 
the  following  conditions  were  true),  then  the  preceding  steps  were  repeated  separately  for  two 
groups,  one  consisting  of  Catholic  schools  and  one  consisting  of  non-Catholic  schools. 

Conditions  ft  **  Method  2: 

• more  than  10  nonpublic  noncertainty  schools;  and 

• more  than  I Catholic  noncertainty  school;  and 

• more  than  1 non-Catholic  noncertainty  school. 

For  jurisdictions  with  certainty  nonpublic  schools  (Delaware,  District  of  Columbia,  and 
Montana  at  grade  4,  and  Delaware,  District  of  Columbia,  Maryland,  North  Dakota,  Rhode  Island, 
Vermont,  and  Wyoming  at  grade  8)  each  school  was  assigned  to  one  or  more  groups.  If  a school 
was  selected  three  or  more  times  in  the  sampling,  it  was  assigned  to  two  groups.  Prior  to  this 
assignment,  schools  were  sorted  in  descending  order  of  the  estimated  grade  enrollment.  The 
group  numbering  started  at  the  last  number  where  the  noncertainty  nonpublic  schools  ended.  A 


replicate  was  formed  by  randomly  deleting  one  half  of  the  students  in  a certain  school  from  the 
sample.  For  the  certainty  schools  that  were  assigned  to  two  replicate  groups,  the  students  were 
split  equally  between  four  “halves,”  two  halves  in  each  of  two  replicate  groups.  This  was 
repeated  for  each  certainty  school. 

Again,  the  aim  was  to  maximize  the  number  of  degrees  of  freedom  for  estimating 
sampling  errors  for  nonpublic  schools  (and  indeed  for  public  and  nonpublic  schools  combined) 
within  the  constraint  of  forming  62  replicate  groups.  Where  a jurisdiction  had  a significant 
contribution  from  both  Catholic  and  non-Catholic  schools,  Westat  ensured  that  the  sampling 
error  estimates  reflected  the  stratification  on  this  characteristic. 

Guam.  For  Guam,  where  all  schools  were  selected  with  certainty,  schools  were  assigned 
to  one  or  more  replicate  groups  proportional  to  their  estimated  grade  enrollment. 

DDESS  and  DoDDS  Schools.  Schools  in  the  DDESS  grade  8 sample  were  assigned  to 
one  or  more  replicate  groups  proportional  to  their  estimated  grade  enrollment.  Schools  in  all 
other  DDESS  and  DoDDS  samples  were  assigned  to  replicate  groups  following  the  general  rules 
described  above  for  all  public  schools. 


7.6.2  School-Level  Replicate  Weights 

As  mentioned  above,  each  replicate  sample  had  to  be  reweighted  to  compensate  for  the 
dropped  unit(s)  defining  the  replicate.  This  reweighting  was  done  in  two  stages.  At  the 
first-stage,  the  ith  school  included  in  a particular  replicate  r was  assigned  a replicate- specific 
school  base  weight  defined  as: 


wx;,h  = Kr  x Wf 

where  W*h  is  the  full-sample  base  weight  for  school  i,  and,  for  public  schools: 


1.5  if  school  i was  contained  in  a "pair"  consisting  of  3 units  from  which 
the  complementary  member  was  dropped  to  form  replicate  r, 


K, 


2 if  school  i was  contained  in  a pair  consisting  of  2 units  from  which  the 
complementary  member  was  dropped  to  form  replicate  r. 


0 if  school  i was  dropped  to  form  replicate  r,  and 

1 if  school  / was  not  assigned  to  replicate  r or  if  school  i was  a certainty. 
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For  nonpublic  schools,  Method  1 : 


Kr  = \n- 1 


n 


if  school  i was  not  dropped  in  forming  replicate  r 


[0  if  school  i was  dropped  to  form  replicate  r 


For  nonpublic  schools,  Method  2 (with  nt  Catholic  schools  and  n2  non-Catholic  schools): 

h,  if  school  i was  Catholic,  not  dropped  from  replicate  r, 
n\  - 1 and  replicate  r was  formed  by  dropping  a Catholic  school 

1 if  school  i was  Catholic  and  replicate  r was  formed  by  dropping  a nonCatholic  school 

Kr  =J 

ti2  if  school  i was  nonCatholic,  not  dropped  from  replicate  r, 
ri2  - 1 and  replicate  r was  formed  by  dropping  a nonCatholic  school 

1 if  school  i was  nonCatholic  and  replicate  r was  formed  by  dropping  a Catholic  school 

0 if  school  i was  dropped  to  form  replicate  r 


Using  the  replicate-specific  school  base  weights,  Wn'ch,  the  school-level  nonresponse 
weighting  adjustments  were  recalculated  for  each  replicate  r.  That  is,  the  school-level 
nonresponse  adjustment  factor  for  schools  in  replicate  r and  adjustment  class  k was  computed  as: 

XW'  x Eh) 

Frk  ' IW  x Eh  x srkl) 

'6C, 


where 


Q = the  subset  of  school  records  in  adjustment  class  k\ 

Wit*  ” the  replicate-r  base  weiglit  of  the  iih  school  in  class  k\ 

Eh  = the  grade  enrollment  for  the  /th  school  in  class  k\ 

In  the  above  formulation,  the  indicator  variable  8rki  a nonzero  value  only  when  the  /th 
school  in  replicate  r and  adjustment  class  k participated  in  the  assessment.  The  replicate-specific 
nonresponse-adjusted  school  weight  for  the  /th  school  in  replicate  r in  class  k was  then  computed 


W“!i  = F rk  X w%  X 8r«  ■ 


7.6.3  Student-Level  Replicate  Weights 

The  replicate-specific  adjusted  student  base  weights  were  calculated  by  multiplying  the 
replicate-specific  adjusted  school  weights  as  described  above  by  the  corresponding  within-school 
student  weights.  That  is,  the  adjusted  student  base  weight  for  the  y'th  student  in  adjustment  class  k 
in  replicate  r was  initially  computed  as: 

W rkij  = x Wf'in 


where 


Wrki  = the  nonresponse-adjusted  school  weight  for  school  i in  school 
adjustment  class  k and  replicate  r;  and 

Within  _ the  within-school  weight  for  the  jth  student  in  school  i. 

The  final  replicate-specific  student  weights  were  then  obtained  by  applying  the  student 
nonresponse  adjustment  procedures  to  each  set  of  replicate  student  weights.  Let  Frj<  denote  the 
student-level  nonresponse  adjustment  factor  for  replicate  r and  adjustment  class  k . The  final 
replicate-r  student  weight  for  student  j in  school  i in  adjustment  class  k was  calculated  as: 

W$*  = F rk  X W*H  X wfhin  . 


Finally,  estimates  of  the  variance  of  sample-based  estimates  were  calculated  as: 

62 

VarjK(-*)  ~ HO,  (*r  ' * F > 

r=I 

where 

v - X x xuj 

i*j 

denotes  an  estimated  total  based  on  the  full  sample,  and  xr  denote  the  corresponding  estimate 
based  on  replicate  r with  62  replicates.  The  standard  error  of  an  estimate  x is  estimated  by  taking 
the  square  root  of  the  estimated  variance,  VariK(x). 
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7.7  RAKING  OF  WEIGHTS 


Raking  (also  known  as  iterative  proportional  fitting)  is  done  in  place  of  poststratification. 
Unlike  poststratification,  it  is  performed  iteratively  to  two  or  more  different  distributions  of  a 
population  total  (i.e.,  gender  and  age).  It  is  typically  used  in  situations  in  which  the  interior  cells 
of  a cross-tabulation  are  either  unknown,  or  some  sample  sizes  in  the  cells  are  too  small  for 
efficient  estimation.  In  raking,  the  marginal  population  totals,  /V,-.  and  Nj  are  known  (i.e.,  age  and 
gender  population  counts),  however,  the  interior  cells  of  the  cross-tabulation  Ntj  (the  age  by 
gender  cells)  are  estimated  from  the  sample  by  , where  these  are  the  sum  of  weights  in  the 

cells.  The  raking  algorithm  proceeds  by  proportionally  scaling  the  fj- , such  that  the  following 

relations  are  satisfied: 


= Nl 
j 

and 

XM,  = Nj. 

i 

The  1996  State  Assessment  program  used  two  different  sets  of  inclusion  rules  indicated 
by  sample  type  = 1 and  sample  type  = 2 (see  Chapter  3).  The  mathematics  assessment  was 
analyzed  omitting  the  sample  type  = 2 SD/LEP  students;  and  the  science  assessment  was 
analyzed  omitting  the  sample  type  = 1 SD/LEP  students.  The  SD/LEP  student  weights  were 
raked  separately  for  the  two  subsets  as  defined  by  sample  type  and  public/nonpublic  schools. 
Agreement  was  forced  with  totals  estimated  using  both  of  the  subsets  combined  for  each  of  the 
school  types.  The  purpose  of  this  was  to  enhance  the  reliability  (i.e.,  reduce  the  sampling  error) 
of  estimates  produced  by  using  information  about  student  characteristics  from  the  whole  sample 
to  enhance  the  estimates. 


7.7.1  Raking  Dimensions  for  Full  Sample  Student  Weights 

Public  Schools.  Five  variables  were  used  for  the  raking  dimensions.  These  variables 
included  two  levels  of  SD  (SD/nonSD),  two  levels  of  LEP  (LEP/nonLEP),  two  levels  of 
GENDER,  five  levels  of  RACE  (White  and  Other;  Black;  Hispanic;  Asian  or  Pacific  Islander; 
and  American  Indian  or  Alaskan  Native),  and  two  levels  of  AGE.  The  variable  AGE  was  defined 
as  follows:  for  grade  4,  those  bom  in  August  1985  or  earlier  and  those  bom  in  September  1985  or 
later  and  for  grade  8,  those  bom  in  August  1981  or  earlier  and  those  bom  in  September  1981  or 
later.  Collapsing  of  levels  was  done  so  that  no  level  of  a single  dimension  contained  fewer  than 
30  students  for  a state  and  grade. 

Tables  7-15  and  7-16  show  for  each  jurisdiction  and  grade,  the  final  collapsed  levels  that 
were  used  for  the  raking  dimensions.  A dash  indicates  that  all  levels  were  combined,  and  thus, 
the  variable  was  not  used  as  a raking  dimension.  An  asterisk  for  the  RACE  variable  indicates  that 
all  other  levels  of  the  dimension  were  combined  into  one  level.  For  example  in  fourth  grade  for 
Alaska,  there  are  four  levels  of  race:  White,  Hispanic,  American  Indian  or  Alaskan  Native,  and 
all  others  combined. 


Nonpublic  Schools.  Because  of  the  small  numbers  of  nonpublic- school  students,  no 
raking  was  carried  out.  A factor  of  2 was  applied  to  the  weights  for  the  SD/LEP  students  since 
only  half  the  SD/LEP  sample  was  used  for  analysis. 


7.7.2  Raking  Student  Replicate  Weights 

The  replicate  weights  for  the  public  SD/LEP  students  were  raked  similarly.  Control 
totals  for  each  replicate  were  calculated  based  on  the  totals  for  the  replicate  weights.  The  levels 
of  the  raking  dimensions  that  were  used  for  the  replicates  were  the  same  collapsed  levels  as 
shown  in  Tables  7-15  and  7-16.  For  the  nonpublic  schools,  again  a factor  of  2 was  applied  to  the 
replicate  weights  of  the  SD/LEP  students. 


(■ 
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Table  7-15 

Final  Collapsed  Levels  U sed  for  Raking  Dimensions ; Grade  4 


Jurisdiction 


Alabama 

Gender 

Age 

Race:  W /* 

- 

- 

Alaska 

Gender 

Age 

Race:  W/H/N/* 

SD 

LEP 

Arizona 

Gender 

Age 

Race:  H /* 

SD 

LEP 

Arkansas 

Gender 

Age 

Race:  W / * 

- 

- 

California 

Gender 

Age 

Race:  W/H/* 

SD 

LEP 

Colorado 

Gender 

Age 

Race:  W/H/* 

SD 

LEP 

Connecticut 

Gender 

Age 

Race:  W / H / * 

SD 

LEP 

Delaware 

Gender 

- 

Race:  W/B/* 

- 

- 

District  of  Columbia 

Gender 

Age 

Race:  B / * 

SD 

- 

Florida 

Gender 

Age 

Race:  W/H/* 

SD 

LEP 

Georgia 

Gender 

Age 

Race:  W/B/* 

- 

- 

Hawaii 

Gender 

- 

Race:  H / A / * 

SD 

LEP 

Indiana 

Gender 

Age 

Race:  W / * 

- 

- 

Iowa 

Gender 

Age 

Race:  W /* 

- 

- 

Kentucky 

Gender 

Age 

- 

- 

- 

Louisiana 

Gender 

Age 

Race:  B / * 

- 

- 

Maine 

Gender 

Age 

- 

- 

- 

Maryland 

Gender 

- 

Race:  W /* 

- 

- 

Massachusetts 

Gender 

Age 

Race:  W /* 

SD 

- 

- Michigan 

Gender 

Age 

Race:  W /* 

- 

- 

Minnesota 

Gender 

Age 

Race:  W / * 

SD 

- 

Mississippi 

Gender 

Age 

Race:  B /* 

- 

- 

Missouri 

Gender 

Age 

Race:  W /* 

- 

- 

Montana 

Gender 

Age 

- 

- 

- 

Nebraska 

Gender 

Age 

Race:  W/H/* 

- 

- 

Nevada 

Gender 

Age 

Race:  H /* 

SD 

LEP 

New  Jersey 

Gender 

Age 

Race:  W /* 

- 

- 

New  Mexico 

Gender 

Age 

Race:  W / H / * 

SD 

LEP 

New  York 

Gender 

Age 

Race:  W /* 

SD 

- 

North  Carolina 

Gender 

Age 

Race:  W/B/* 

- 

- 

North  Dakota 

Gender 

Age 

- 

- 

- 

Oregon 

Gender 

Age 

Race:  W /* 

SD 

LEP 

Pennsylvania 

Gender 

Age 

- 

- 

- 

Rhode  Island 

Gender 

Age 

Race:  W/* 

SD 

LEP 

South  Carolina 

Gender 

Age 

Race:  B /* 

- 

- 

Tennessee 

Gender 

Age 

Race:  W/* 

- 

- 

Texas 

Gender 

Age 

Race:  W/H/* 

SD 

LEP 

Utah 

Gender 

Age 

Race:  W / * 

- 

LEP 

Vermont 

Gender 

Age 

- 

- 

- 

Virginia 

Gender 

Age 

Race:  W/B/* 

- 

- 

Washington 

Gender 

Age 

Race:  W / * 

SD 

- 

West  Virginia 

Gender 

Age 

- 

- 

- 

Wisconsin 

Gender 

Age 

Race:  W /* 

- 

* 

Wyoming 

Gender 

Age 

Race:  W /* 

- 

- 

Other  jurisdictions 

Guam 

Gender 

- 

- 

SD 

LEP 

DoDE  A/DDRS  S 

- 

- 

- 

- 

- 

DoDEA/DoDDS 

Gender 

Age 

Race:  W /* 

- 

* 

Table  7-16 

Final  Collapsed  Levels  Used  for  Raking  Dimensions , Grade  8 


Jurisdiction 


Alabama 

Gender 

Age 

Race:  W/* 

- 

- 

Alaska 

Gender 

Age 

Race:  W / * 

- 

- 

Arizona 

Gender 

Age 

Race:  H/  * 

SD 

LEP 

Arkansas 

Gender 

Age 

Race:  W / * 

- 

- 

California 

Gender 

Age 

Race:  W/H/* 

SD 

LEP 

Colorado 

Gender 

Age 

Race:  W / * 

- 

- 

Connecticut 

Gender 

Age 

Race:  W/* 

- 

- 

Delaware 

Gender 

Age 

Race:  W/* 

- 

- 

District  of  Columbia 

Gender 

Age 

Race:  B / * 

SD 

- 

Florida 

Gender 

Age 

Race:  W/H/* 

SD 

- 

Georgia 

Gender 

Age 

Race:  W / * 

- 

- 

Hawaii 

Gender 

- 

Race:  A / * 

SD 

LEP 

Indiana 

Gender 

Age 

Race:  W/* 

- 

- 

Iowa 

Gender 

Age 

- 

- 

- 

Kentucky 

Gender 

Age 

- 

- 

- 

Louisiana 

Gender 

Age 

Race:  B / * 

- 

- 

Maine 

Gender 

Age 

- 

- 

- 

Maryland 

Gender 

Age 

Race:  B / * 

- 

- 

Massachusetts 

Gender 

Age 

Race:  W /* 

- 

- 

Michigan 

Gender 

Age 

- 

- 

- 

Minnesota 

Gender 

Age 

- 

- 

- 

Mississippi 

Gender 

Age 

Race:  B / * 

- 

- 

Missouri 

Gender 

Age 

Race:  W/* 

- 

- 

Montana 

- 

Age 

- 

- 

- 

Nebraska 

Gender 

Age 

Race:  W/* 

- 

- 

Nevada 

- 

- 

- 

- 

- 

New  Hampshire 

Gender 

Age 

- 

- 

- 

New  Jersey 

Gender 

Age 

Race:  W/* 

- 

- 

New  Mexico 

Gender 

Age 

Race:  W / H / * 

SD 

LEP 

New  York 

Gender 

Age 

Race:  W/H/* 

SD 

- 

North  Carolina 

Gender 

Age 

Race:  W /* 

- 

- 

North  Dakota 

Gender 

Age 

- 

- 

- 

Oregon 

Gender 

Age 

Race:  W/* 

- 

- 

Rhode  Island 

Gender 

Age 

Race:  W / * 

SD 

- 

South  Carolina 

Gender 

Age 

Race:  B / * 

- 

- 

Tennessee 

Gender 

Age 

Race:  W / * 

- 

- 

Texas 

Gender 

Age 

Race:  H / * 

SD 

LEP 

Utah 

Gender 

Age 

- 

- 

- 

Vermont 

Gender 

Age 

- 

- 

- 

Virginia 

Gender 

Age 

Race:  W/* 

- 

Washington 

Gender 

Age 

Race:  W/* 

- 

LEP 

West  Virginia 

Gender 

Age 

- 

- 

- 

Wisconsin 

Gender 

Age 

Race:  W / * 

- 

- 

Wyoming 

Gender 

Age 

- 

- 

- 

Other  jurisdictions 

Guam 

- 

- 

- 

- 

- 

DoDEA/DDESS 

- 

- 

- 

- 

- 

DoDEA/DoDDS 

- 

Race:  W / * 

- 

- 
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Chapter  8 


THEORETICAL  BACKGROUND  AND  PHILOSOPHY  OF  NAEP 

SCALING  PROCEDURES1 

Eugene  G.  Johnson  and  Nancy  L.  Allen 
Educational  Testing  Service 


8.1  OVERVIEW 

The  primary  method  by  which  results  from  the  State  Assessment  are  disseminated  is 
scale-score  reporting.  With  scaling  methods,  the  performance  of  a sample  of  students  in  a subject 
area  or  subarea  can  be  summarized  on  a single  scale  or  a series  of  scales  even  when  different 
students  have  been  administered  different  items.  This  chapter  presents  an  overview  of  the  scaling 
methodologies  employed  in  the  analyses  of  the  data  from  NAEP  surveys  in  general  and  from  the 
State  Assessment  in  mathematics  in  particular.  Details  of  the  scaling  procedures  specific  to  the 
1996  State  Assessment  in  mathematics  are  presented  in  Chapter  9. 


8.2  BACKGROUND 

The  basic  information  from  an  assessment  consists  of  the  responses  of  students  to  the 
items  presented  to  them.  For  NAEP,  these  items  are  constructed  to  measure  performance  on  sets 
of  objectives  developed  by  nationally  representative  panels  of  learning  area  specialists, 
educators,  and  concerned  citizens.  Satisfying  the  framework  and  specifications  for  the 
assessment  and  ensuring  that  the  items  selected  to  measure  each  part  of  the  framework  cover  a 
range  of  difficulty  levels  typically  requires  many  items.  For  example,  the  State  Assessment  in 
mathematics  required  104  items  at  grade  4 only,  129  items  at  grade  8 only,  and  54  items  at  both 
grades  4 and  8 to  meet  the  specifications  provided  for  the  assessment.  To  reduce  student  burden, 
each  assessed  student  was  presented  only  a fraction  of  the  full  pool  of  items  through  multiple 
matrix  sampling  procedures. 

The  most  direct  manner  of  presenting  the  assessment  results  is  to  report  separate  results 
for  each  item.  However,  because  of  the  vast  amount  of  information,  having  separate  results  for 
each  of  the  items  in  the  assessment  pool  hinders  our  understanding  of  the  overall  performance  of 
subgroups  of  the  population.  Item-by-item  reporting  masks  our  understanding  of  similarities  in 
trends  and  subgroup  comparisons  common  across  items. 

An  obvious  way  to  summarize  performance  across  a collection  of  items  is  to  calculate 
the  average  of  the  separate  item  scores.  The  advantage  of  averaging  is  that  it  tends  to  cancel  out 
the  effects  of  peculiarities  in  items  that  can  affect  item  difficulty  in  unpredictable  ways. 
Furthermore,  averaging  makes  it  easier  to  compare  the  general  performances  of  subpopulations. 


’ Nancy  L.  Allen  was  responsible  for  the  psychometric  and  statistical  analysis  of  state  and  national  NAEP  data.  Eugene 
G.  Johnson  is  a senior  psychomctrician  with  special  expertise  in  the  design  of  NAEP  and  sampling  issues  Robert  J. 
Mislcvy  and  Neal  Thomas  contributed  greatly  to  previous  versions  of  this  chapter. 
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Despite  their  advantages,  there  are  a number  of  significant  problems  with  average  item 
scores.  First,  the  interpretation  of  these  averages  depends  on  the  items  that  happen  to  be 
administered  to  a group  of  students.  Since  all  students  are  not  administered  the  same  items,  the 
average  item  score  of  students  who  happen  to  be  administered  a set  of  easy  or  difficult  items 
would  make  that  group’s  performance  appear  to  be  overly  high  or  low.  Second,  again  since  the 
average  score  is  related  to  the  particular  items  administered  direct  comparisons  of  subpopulations 
become  difficult  because  they  require  that  those  subpopulations  be  administered  the  same  set  of 
items.  Third,  because  this  approach  limits  comparisons  to  average  scores  on  specific  sets  of 
items,  it  provides  no  simple  way  to  report  trends  over  time  when  the  specific  content  of  the  item 
pool  changes.  Finally,  direct  estimates  of  quantities  such  as  the  proportion  of  students  who  would 
achieve  a specific  score  across  the  items  in  the  pool  are  not  possible  when  every  student  is 
administered  only  a small  fraction  of  the  item  pool.  While  the  mean  score  across  all  items  in  the 
pool  can  be  readily  obtained  (by  calculating  the  average  of  the  individual  item  scores),  statistics 
that  provide  distributional  scores  across  the  full  set  of  items  in  the  pool  cannot  be  readily 
obtained  without  additional  assumptions. 

These  limitations  can  be  overcome  by  the  use  of  response  scaling  methods.  When  several 
items  require  similar  skills,  the  regularities  observed  in  response  patterns  can  often  be  used  to 
characterize  both  students  and  items  using  a relatively  small  number  of  variables.  These  variables 
include  a student-specific  variable,  commonly  called  proficiency,  estimated  by  the  scale  score, 
which  quantifies  a student’s  tendency  to  answer  items  correctly  (or,  for  multipoint  items,  to 
achieve  a certain  score)  and  item-specific  characteristics  of  an  item  such  as  its  difficulty, 
effectiveness  in  distinguishing  between  students  with  different  levels  of  proficiency,  and  chances 
of  a very  low  proficiency  student  correctly  answering  a multiple-choice  item.  (These  variables 
are  discussed  in  more  detail  in  the  next  section.)  When  combined  through  appropriate 
mathematical  formulas,  these  characteristics  capture  the  dominant  features  of  the  data. 
Furthermore,  all  students’  proficiencies  can  be  measured  on  a common  scale,  even  though  none 
of  the  students  took  all  of  the  items  in  the  pool.  Using  the  common  scale,  it  becomes  possible  to 
estimate  distributions  of  proficiency  in  a population  or  subpopulation  and  to  estimate  the 
relationships  between  the  scale  scores  and  student  background  variables. 

It  is  important  to  point  out  that  any  procedure  of  aggregation,  whether  it  be  a simple 
average  or  a more  complex  multidimensional  scaling  model,  highlights  certain  patterns  at  the 
expense  of  other  potentially  interesting  patterns  that  may  reside  within  the  data.  Every  item  in  a 
NAEP  survey  is  of  interest  and  can  provide  useful  information  about  what  young  Americans 
know  and  can  do.  The  choice  of  an  aggregation  procedure  must  be  driven  by  a conception  of 
which  patterns  are  most  important  for  a particular  purpose. 

The  scaling  for  the  State  Assessment  in  mathematics  was  carried  out  separately  for  each 
of  the  five  mathematics  content  strands  specified  in  the  framework  for  grade  4 mathematics  and 
then  for  grade  8 mathematics.  This  scaling  within  subareas  was  done  because  it  was  anticipated 
that  different  patterns  of  performance  might  exist  for  these  essential  subdivisions  of  the  subject 
area.  The  five  content  strands  are:  number  sense , properties , and  operations ; measurement ; 
geometry  and  spatial  sense ; data  analysis ; statistics , and  probability ; and  algebra  and  functions. 
By  creating  a separate  scale  for  each  of  these  content  strands,  any  differences  in  subpopulation 
performance  between  the  content  strands  are  preserved.  Comparisons  of  the  difficulty  of  one 
subarea  with  another  are  not  possible  with  scale  results.  Average  item  scores  for  the  different 
subareas  could  be  compared,  but  these  average  scores  are  related  to  the  particular  items  in  the 
assessment,  rather  than  to  any  innate  characteristics  of  items  within  any  subarea. 


The  creation  of  a series  of  separate  scales  to  describe  mathematics  performance  does  not 
preclude  the  reporting  of  a single  index  of  overall  mathematics  performance — that  is,  an  overall 
mathematics  composite.  A composite  is  computed  as  the  weighted  average  of  the  five  content 
strand  scales,  where  the  weights  correspond  to  the  relative  importance  given  to  each  content 
strand  as  defined  by  the  framework.  The  composite  provides  a global  measure  of  performance 
within  mathematics,  whereas  the  content  strand  scales  allow  the  measurement  of  important 
interactions  among  these  subdivisions  of  mathematics. 


8.3  SCALING  METHODOLOGY 

This  section  reviews  the  scaling  models  employed  in  the  analyses  of  data  from  the  State 
Assessment  in  mathematics  and  the  1996  national  mathematics  assessment.  It  also  reviews  the 
multiple  imputation  or  “plausible  values”  methodology  that  allows  such  models  to  be  used  with 
NAEP’s  sparse  item-sampling  design.  The  reader  is  referred  to  Mislevy  (1991)  for  an 
introduction  to  plausible  values  methods  and  a comparison  with  standard  psychometric  analyses, 
to  Mislevy,  Johnson,  and  Muraki  (1992)  and  Beaton  and  Johnson  (1992)  for  additional 
information  on  how  the  models  are  used  in  NAEP,  and  to  Rubin  (1987)  for  the  theoretical 
underpinnings  of  the  approach.  It  should  be  noted  that  the  imputation  procedure  used  by  NAEP  is 
a mechanism  for  providing  plausible  values  for  scale  score  averages  and  not  for  filling  in  blank 
responses  to  background  or  cognitive  variables. 

While  the  NAEP  procedures  were  developed  explicitly  to  handle  the  characteristics  of 
NAEP  data,  they  build  on  work  paralleled  by  other  researchers.  See,  for  example  Dempster, 
Laird,  and  Rubin  (1977);  Little  and  Rubin  (1983,  1987);  Andersen  (1980);  Engelen  (1987); 
Hoijtink  (1991);  Laird  (1978);  Lindsey,  Clogg,  and  Grego  (1991);  Zwinderman  (1991);  Tanner 
and  Wong  (1987);  and  Rubin  (1987,  1991). 

The  104  mathematics  items  administered  exclusively  at  grade  4,  the  129  items 
administered  exclusively  at  grade  8,  and  the  54  items  administered  at  both  grades  4 and  8 in  the 
State  Assessment  were  also  administered  to  students  of  the  same  grades  in  the  national 
mathematics  assessment.  The  number  of  items  actually  scaled  differs  from  the  number  of  items 
administered  due  to  decisions  about  the  treatment  of  items  in  scaling  (see  Tables  9-1  through  9-4 
in  Chapter  9).  However,  because  the  administration  procedures  differed,  the  State  Assessment 
data  were  scaled  independently  from  the  national  data.  The  national  assessment  also  included 
results  for  students  in  grade  12;  and  results  on  an  estimation  scale  for  each  grade.  Details  of  the 
scaling  of  the  State  Assessment  and  the  subsequent  linking  to  the  results  from  the  national 
mathematics  assessment  are  provided  in  Chapter  9. 


8.3.1  The  Scaling  Models 

Three  distinct  scaling  models,  depending  on  item  type  and  scoring  procedure,  were  used 
in  the  analysis  of  the  data  from  the  State  Assessment.  Each  of  the  models  is  based  on  item 
response  theory  (IRT;  e.g..  Lord,  1980).  Each  is  a “latent  variable”  model,  which  is  defined 
separately  for  each  of  the  scales  at  each  grade.  A latent  variable  model  expresses  students’ 
tendencies  to  respond  (such  as  correct/incorrect)  on  the  items  as  a function  of  a characteristic  that 
is  not  directly  observed.  This  characteristic  is  called  proficiency.  Students’  proficiencies  are 
estimated  by  scale  scores. 
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A three-parameter  logistic  (3PL)  model  was  used  for  the  multiple-choice  items  (which 
were  scored  correct/incorrect).  The  fundamental  equation  of  the  3PL  model  is  the  probability  that 
a student,  whose  proficiency  on  scale  k is  characterized  by  the  unobservable  variable  will 
respond  correctly  to  item  j: 


P(xj  = 1|  OkMj,bj,Cj)-c} 


(l-o) 


l+exp[-1.7a,-(ft-fe/)] 


= Pj  i(ft) 


(8.1) 


In  Equation  8.1 : 

xj  is  the  response  to  item  j , 1 if  correct  and  0 if  not; 

a,  where  a}>  0,  is  the  slope  parameter  of  item  y,  characterizing  the  strength  of  its 

relationship  to  the  latent  proficiency; 

bj  is  the  location  parameter  of  item  y.  characterizing  its  difficulty  with  respect  to  the 
latent  proficiency;  and 

Cj  where  0 < q<  1,  is  the  lower  asymptote  parameter  of  item y,  reflecting  the 
chances  of  students  of  very  low  proficiency  selecting  the  correct  option. 

Further,  the  probability  of  an  incorrect  response  to  the  item  is  defined  as: 

Pjo  = P(xj  = 0\ek,aj,bj,cj)  = J - Pj,(0k)-  (8.2) 

A two-parameter  logistic  (2PL)  model  was  used  for  short  constructed-response  items, 
which  were  scored  correct  or  incorrect.  The  equations  of  the  2PL  model  are  the  same  as  those  of 
Equations  8. 1 and  8.2,  with  the  q parameter  fixed  at  zero. 

In  addition  to  the  multiple-choice  and  short  constructed-response  items,  a number  of 
extended  constructed-response  items  (5  at  grade  4 and  6 at  grade  8)  were  administered  in  the 
State  and  national  assessments.  Each  of  these  items  was  scored  on  a multipoint  ranging  from  0 to 
4.  Additionally,  as  discussed  in  Chapter  9,  certain  sets  of  items  consisting  of  highly  correlated 
parts  were  combined  into  cluster  items  or  “testlets”  (Wainer  & Kiely,  1987)  where  the  score 
assigned  to  a cluster  item  was  the  number  of  constituent  parts  answered  correctly.  Items  which 
are  scored  on  a multipoint  scale  are  referred  to  as  polytomous  items.  The  multiple-choice  and 
short  constructed-response  items,  which  are  scored  correct/incorrect,  are  referred  to  as 
dichotomous  items. 

The  polytomous  items  were  scaled  using  a generalized  partial  credit  model  (Muraki, 

1992).  The  fundamental  equation  of  this  model  is  the  probability  that  a person,  whose 
proficiency  on  scale  k is  characterized  by  the  unobservable  variable  9k , will  respond  to  item  j in  a 
way  to  be  scored  /: 


i 


hi 
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exp(£l.7a,(&  - bj  + dj.  v) 


P(  Xj  /I  dk.cij,  bj.dj.i,...,  dj.mri)  mj~] 


v=0 


= />,,(&) 


Sexp  ]£l.7 aj((k-bj  + dj.i) 

g= o V V=0 


(8.3) 


where 


rrij  is  the  number  of  ordered  categories  in  response  to  item; 

Xj  is  the  response  to  item  j,  with  possibilities  i = 0,1, 1 
a}  is  the  slope  parameter; 

bj  is  the  item  location  parameter  characterizing  overall  difficulty  with  respect  to  the 
latent  proficiency;  and 

dj,!  is  the  category  i threshold  parameter  (see  below). 

Indeterminacies  in  the  parameters  of  the  above  model  are  resolved  by  setting  dji0  = 0 and  setting 

ntj’l 

£ dj.i  = 0.  Muraki  (1992)  points  out  that  bj  - dJA  is  the  point  on  the  6k  scale  at  which  the  plots 

i=i 

of  Pj,i-i(0k)  and  Pji(&k)  intersect  and  so  characterizes  the  point  on  the  6k  scale  at  which  the 
student’s  response  to  item  j has  equal  probability  of  falling  in  score  category  i - 1 and  falling  in 
score  category  i. 


When  m;  = 2,  so  that  there  are  two  score  categories  (such  as  0,1),  it  can  be  shown  that 
Pj,(( %)  of  Equation  8.3  for  z'=0,l  corresponds  respectively  to  Pjd  6k)  and  P,i(  6k)  of  the  2PL  model 
(Equations  8.1  and  8.2  with  cp 0). 

A typical  assumption  of  item  response  theory  is  the  independence  of  the  response  by  a 
student  to  a set  of  items,  given  or  conditional  on  the  student’s  proficiency.  That  is,  for  a student 
with  a specific  proficiency  of  6k,  the  joint  probability  of  a particular  response  pattern 
x = (x,,...jcn)  across  a set  of  n items  is  simply  the  product  of  terms  based  on  Equations  8.1, 8.2, 
and  8.3: 

j>  zd. 

P(x\9k,  item  parameters)  = li  O Pj,(0k)u'' 

j=l  i=0 

where  P]t(  6k)  is  of  the  form  appropriate  to  the  type  of  item  (dichotomous  or  polytomous),  my  is 
taken  equal  to  2 for  the  dichotomously  scored  items,  and  Ky,  is  an  indicator  variable  defined  by 

1 if  response  jo  was  in  score  category  i 
uji  = L , 

0 otherwise 
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It  is  also  typically  assumed  that  response  probabilities  are  conditionally  independent  of 
background  variables  (yj,  given  0kj  or 


P(x\Qk,item  parameters, y)  = p(x\Qk, item  parameters). 

After  x has  been  observed,  Equation  8.4  can  be  viewed  as  a likelihood  function,  and 
provides  a basis  for  inference  about  6k  or  about  item  parameters.  Estimates  of  item  parameters 
(see  Appendix  D)  were  obtained  by  the  NAEP  BILOG/PARSCALE  program,  which  combines 
Mislevy  and  Bock’s  (1982)  BILOG  and  Muraki  and  Bock’s  (1991)  PARSCALE  computer 
programs,  and  which  concurrently  estimates  parameters  for  all  items  (dichotomous  and 
polytomous).  The  item  parameters  are  then  treated  as  known  in  subsequent  calculations.  The 
parameters  of  the  items  constituting  each  of  the  separate  scales  were  estimated  independently  of 
the  parameters  of  the  other  scales.  Once  items  have  been  calibrated  in  this  manner,  a likelihood 
function  for  the  scale  proficiency  0k  is  induced  by  a vector  of  responses  to  any  subset  of 
calibrated  items,  thus  allowing  0*- based  inferences  from  matrix  samples. 

Item  parameter  estimation  was  performed  separately  for  the  grade  4 and  the  grade  8 data. 
As  stated  previously,  item  parameter  estimation  was  performed  independently  for  the  State 
Assessment  and  for  the  national  mathematics  assessment.  In  both  cases,  the  identical  scale 
definitions  were  used. 

In  all  NAEP  IRT  analyses,  missing  responses  at  the  end  of  each  block  of  items  a student 
was  administered  were  considered  “not-reached,”  and  treated  as  if  they  had  not  been  presented  to 
the  student.  Missing  responses  to  dichotomous  items  before  the  last  obsen'ed  response  in  a block 
were  considered  intentional  omissions,  and  treated  as  fractionally  correct  and  assigned  a score 
equal  to  the  reciprocal  of  the  number  of  response  alternatives.  These  conventions  are  discussed 
by  Mislevy  and  Wu  (1988).  With  regard  to  the  handling  of  not-reached  items,  Mislevy  and  Wu 
found  that  ignoring  not-reached  items  introduces  slight  biases  into  item  parameter  estimation. 

The  degree  of  this  bias  depends  on  the  number  of  not-reached  items  and  whether  speed  is 
correlated  with  ability.  With  regard  to  omissions,  they  found  that  the  method  described  above 
provides  consistent  limited-information  likelihood  estimates  of  item  and  ability  parameters  under 
the  assumption  that  students  omit  only  it  they  can  do  no  better  than  responding  randomly. 

Because  the  extended  constructed-response  items  were  always  the  last  item  in  a block 
and  because  considerably  more  effort  was  required  of  the  student  to  answer  these  items, 
nonresponse  to  an  extended  constructed-response  item  was  considered  an  intentional  omission 
(and  scored  as  the  lowest  category,  0);  however,  if  a student  also  did  not  respond  to  the  item 
immediately  preceding  that  item,  the  extended  constructed-response  item  was  considered  not 
reached  and  treated  as  if  it  had  not  been  presented  to  the  student. 

Although  the  IRT  models  are  employed  in  NAEP  only  to  summarize  performance,  a 
number  of  checks  are  made  to  detect  serious  violations  of  the  assumptions  underlying  the  models 
(such  as  conditional  independence).  When  warranted,  remedial  efforts  are  made  to  mitigate  the 
effects  of  such  violations  on  inferences.  These  checks  include  comparisons  of  empirical  and 
theoretical  item  response  functions  to  identify  items  for  which  the  IRT  model  may  provide  a poor 
fit  to  the  data. 
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The  scales  in  NAEP  are  determined  a priori  by  grouping  items  into  the  content  strands 
defined  by  the  frameworks  developed  by  the  National  Assessment  Governing  Board.  A 
proficiency  scale  6k  is  defined  a priori  by  the  collection  of  items  representing  each  scale.  What  is 
important,  therefore,  is  that  the  models  capture  salient  information  in  the  response  data  to 
effectively  summarize  the  overall  performance  on  the  content  strand  of  the  populations  and 
subpopulations  being  assessed  in  the  content  strand.  NAEP  routinely  conducts  differential  item 
functioning  (DIF)  analyses  to  guard  against  potential  biases  in  making  subpopulation 
comparisons  based  on  the  scale  score  distributions. 

The  local  independence  assumption  embodied  in  Equation  8.4  implies  that  item  response 
probabilities  depend  only  on  6k  and  the  specified  item  parameters,  and  not  on  the  position  of  the 
item  in  the  booklet,  the  content  of  other  items  near  an  item  of  interest,  the  test-administration 
conditions,  or  the  timing  conditions.  However,  these  factors  are  certainly  present  in  any 
administration.  The  practical  question  is  whether  inferences  based  on  the  IRT  probabilities 
obtained  via  Equation  8.4  are  robust  with  respect  to  these  violations  of  the  ideal  assumptions 
underlying  the  IRT  model.  Our  experience  with  the  1986  NAEP  reading  anomaly  (Beaton  & 
Zwick,  1990)  has  shown  that  for  measuring  small  changes  over  time,  changes  in  item  context  and 
speededness  conditions  can  lead  to  unacceptably  large  error  components.  These  can  be  avoided 
by  presenting  items  used  to  measure  change  in  identical  test  forms,  with  identical  timings  and 
administration  conditions.  Thus,  we  do  not  maintain  that  the  item  parameter  estimates  obtained 
in  any  particular  booklet  configuration  are  appropriate  for  other  conceivable  booklet 
configurations.  Rather,  we  assume  that  the  parameter  estimates  are  context-bound.  (For  this 
reason,  we  prefer  common  population  equating  to  common  item  equating  whenever  equivalent 
random  samples  are  available  for  linking.)  This  is  the  reason  that  the  data  from  the  State 
Assessment  were  calibrated  separately  from  the  data  from  the  national  NAEP — since  the 
administration  procedures  diffeied  somewhat  between  the  State  Assessment  and  the  national 
NAEP,  the  values  of  the  item  parameters  could  be  different.  Chapter  9 provides  details  on  the 
procedures  used  to  link  the  results  of  the  1996  State  Assessment  to  those  of  the  1996  national 
assessment  and,  hence,  to  those  of  the  1990  and  1992  Trial  State  and  national  assessments. 


8.3.2  An  Overview  of  Plausible  Values  Methodology 

Item  response  theory  was  developed  in  the  context  of  measuring  individual  students’ 
performance.  In  that  setting,  each  student  is  administered  enough  items  (often  60  or  more)  to 
permit  precise  estimation  of  their  latent  proficiency  6.  This  may  be  accomplished  by  using  a 

A 

maximum  likelihood  estimate  9 , for  example.  When  there  are  enough  items  administered  to  each 
student,  the  uncertainty  associated  with  each  <9  is  negligible.  As  a result,  the  distribution  of  6 , or 
the  joint  distribution  of  tfwith  other  variables,  can  then  be  closely  approximated  using  students’ 

A 

0 values  as  if  they  were  true  lvalues. 


t**  * 

i ( 
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This  approach  breaks  down  in  the  assessment  setting  when,  in  order  to  provide  broader 
content  coverage  in  limited  testing  time,  each  student  is  administered  relatively  few  items  in  a 
scale.  The  problem  is  that  the  uncertainty  associated  with  individual  proficiencies  is  too  large  to 
ignore,  and  the  9 distribution,  as  an  estimate  of  the  ^distribution,  can  be  seriously  biased.  (The 
failure  of  this  approach  was  verified  in  early  analyses  of  the  1984  NAEP  reading  survey;  see 
Wingersky,  Kaplan,  & Beaton,  1987.)  Plausible  value  methodology  was  developed  as  a way  to 
estimate  key  population  features  consistently.  A detailed  development  of  plausible  values 
methodology  is  given  in  Mislevy  (1991).  Along  with  theoretical  justifications,  that  paper  presents 
comparisons  with  standard  procedures,  discussions  of  biases  that  arise  in  some  secondary 
analyses,  and  numerical  examples. 

The  following  provides  a brief  overview  of  the  plausible  values  approach,  focusing  on  its 
implementation  in  the  State  Assessment  analyses. 

Let  y_  represent  the  responses  of  ail  sampled  examinees  to  background  and  attitude 
questions,  along  with  design  variables  such  as  school  membership,  and  let  6 represent  the  vector 
of  scale  proficiency  values.  If  0were  known  for  all  sampled  examinees,  it  would  be  possible  to 
compute  a statistic  t(d,yj — such  as  a scale  or  composite  subpopulation  sample  mean,  a sample 
percentile  point,  or  a sample  regression  coefficient — to  estimate  a corresponding  population 
quantity  T.  A function  U(6,yj — e.g.,  a jackknife  estimate — would  be  used  to  gauge  sampling 
uncertainty,  as  the  variance  of  t around  T in  repeated  samples  from  the  population. 

Because  the  scaling  models  are  latent  variable  models,  however,  lvalues  are  not 
observed  even  for  sampled  students.  To  overcome  this  problem,  we  follow  Rubin  (1987)  by 
considering  6? as  “missing  data’'  and  approximate  t(6>y)  by  its  expectation  given  (x,y),  the  data 
that  actually  were  observed,  as  follows: 

t'(x,X)  = 


= ft(&x)p(&x>X)d£-  (8-6) 

It  is  possible  to  approximate  t*  using  random  draws  from  the  conditional  distribution  of 
the  scale  score  averages  given  the  item  responses  x,,  background  variables  yn  and  model 
parameters  for  sampled  student  /.  These  values  are  referred  to  as  imputations  in  the  sampling 
literature,  and  plausible  values  in  NAEP.  The  value  of  dfov  any  student  that  would  enter  into  the 
computation  of  t is  thus  replaced  by  a randomly  selected  value  from  the  student’s  conditional 
distribution.  Rubin  (1987)  proposes  that  this  process  be  carried  out  several  times — multiple 
imputations — so  that  the  uncertainty  associated  with  imputation  can  be  quantified.  The  average 
of  the  results  of,  M estimates  of  ?,  each  computed  from  a different  set  of  plausible  values,  is  a 
Monte  Carlo  approximation  of  Equation  8.6;  the  variance  among  them,  By  reflects  uncertainty 
due  to  not  observing  0,  and  must  be  added  to  the  estimated  expectation  of  U(6,y)y  which  reflects 
uncertainty  due  to  testing  only  a sample  of  students  from  the  population.  Section  8.5  explains 
how  plausible  values  are  used  in  subsequent  analyses. 

It  cannot  be  emphasized  too  strongly  that  plausible  values  are  not  test  scores  for 
individuals  in  the  usual  sense.  Plausible  values  are  offered  only  as  intermediary  computations  for 
calculating  integrals  of  the  form  of  Equation  8.6,  in  order  to  estimate  population  characteristics. 

When  the  underlying  model  is  correctly  specified,  plausible  values  will  provide  consistent 
estimates  of  population  characteristics,  even  though  they  are  not  generally  unbiased  estimates  of 
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the  scale  score  averages  of  the  individuals  with  whom  they  are  associated.  The  key  idea  lies  in  a 
contrast  between  plausible  values  and  the  more  familiar  d estimates  of  educational  measurement 
that  are  in  some  sense  optimal  for  each  examinee  (e.g.,  maximum  likelihood  estimates,  which  are 
consistent  estimates  of  an  examinee’s  6,  and  Bayes  estimates,  which  provide  minimum  mean- 
squared  errors  with  respect  to  a reference  population):  Point  estimates  that  are  optimal  for 
individual  examinees  have  distributions  that  can  produce  decidedly  nonoptimal  (specifically, 
inconsistent)  estimates  of  population  characteristics  (Little  & Rubin,  1983).  Plausible  values,  on 
the  other  hand,  are  constructed  explicitly  to  provide  consistent  estimates  of  population  effects. 
For  further  discussion  see  Mislevy,  Beaton,  Kaplan,  and  Sheehan  (1992). 


8.3,3  Computing  Plausible  Values  in  IRT-based  Scales 

Plausible  values  for  each  student  r are  drawn  from  the  conditional  distribution 
pt&XnXnr.D),  where  Cand  27 are  regression  model  parameters  defined  in  this  subsection.  This  • 
subsection  describes  how,  in  IRT-based  scales,  these  conditional  distributions  are  characterized, 
and  how  the  draws  are  taken.  An  application  of  Bayes’  theorem  with  the  IRT  assumption  of 
conditional  independence  produces: 

/*£!*,*,  r,S)  <* p(xr idny*  r,S) p(6,\yr,  r,S)  = P(x,\dr) P(dr\^,  r,i i) , (8.7) 

where,  for  vector-valued  6r,  P(xr \dr)  is  the  product  over  scales  of  the  independent  likelihoods 
induced  by  responses  to  items  within  each  scale,  and  p(dr\y^  r,E)  is  the  multivariate — and 
generally  nonindependent— joint  density  of  scale  score  averages  for  the  scales,  conditional  on  the 
observed  value  y,  of  background  responses,  and  the  parameters  f and  27.  The  scales  are 
determined  hy  the  item  parameter  estimates  that  constrain  the  population  mean  to  zero  and 
standard  deviation  to  one.  The  item  parameter  estimates  are  fixed  and  regarded  as  population 
values  in  the  computation  described  in  this  subsection. 

In  the  analyses  of  the  data  from  the  State  Assessment  and  the  data  from  the  national 
mathematics  assessment,  a normal  (Gaussian)  form  was  assumed  for  p( 6r\yn  r,S),  with  a 
common  variance-covariance  matrix,  27,  and  with  a mean  given  by  a linear  model  with  slope 
parameters,  1 ",  based  on  the  first  170  to  271  principal  components  of  870  selected  main  effects 
for  grade  4 and  based  on  the  first  1 52  to  305  principal  components  of  1 ,029  main  effects  for 
grade  8,  and  two-way  interactions  of  the  complete  vector  of  background  variables.  The  variance- 
covariance  matrix,  27,  is  common  across  different  patterns  of  responses  to  the  background 
variable  >>.  The  included  principal  components  will  be  referred  to  as  the  conditioning  variables, 
and  will  be  denoted  . (The  complete  set  of  original  background  variables  used  in  the  State 
Assessment  mathematics  analyses  are  listed  in  Appendix  C.)  The  following  model  was  fit  to  the 
data  within  each  state: 

0 = P y + e . (8-15) 
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where  eis  multivariately  normally  distributed  with  mean  zero  and  variance-covariance  matrix  27. 
The  number  of  principal  components  of  the  conditioning  variables  used  for  each  state  was 
sufficient  to  account  for  90  percent  of  the  total  variance  of  the  full  set  of  conditioning  variables 
(after  standardizing  each  variable).  As  in  regression  analysis,  r is  a matrix  each  of  whose 
columns  contains  the  effects  for  one  scale  and  27is  the  matrix  variance-covariance  of  residuals 
between  scales.  By  fitting  the  model  (Equation  8.8)  separately  within  each  state,  interactions 
between  each  state  and  the  conditioning  variables  are  automatically  included  in  the  conditional 
joint  density  of  scale  score  averages. 

A A 

Maximum  likelihood  estimates  of  /"and  27,  denoted  by  T and  Z,  are  obtained  from  an 
enhancement  of  Sheehan’s  (1985)  MGROUP  computer  program  using  the  EM  algorithm 
described  in  Mislevy  (1985).  The  EM  algorithm  requires  the  computation  of  the  mean,  0.,  and 
variance,  27?,  of  the  posterior  distribution  in  Equation  8.7.  These  moments  are  computed  using 
higher  order  asymptotic  corrections  (Thomas,  1993). 

After  completion  of  the  EM  algorithm,  the  plausible  values  are  drawn  in  a three-step 
process  from  the  joint  distribution  of  the  values  of  /"for  all  sampled  students.  First,  a value  of  r 
is  drawn  from  a normal  approximation  to  P( T,  XI  xn  yr)  that  fixes  27at  the  value  Z , (Thomas, 

1993).  Second,  conditional  on  the  generated  value  of  /"(and  the  fixed  value  of  Z = Z),  the 
mean,  0r,  and  variance,  27?,  of  the  posterior  distribution  in  Equation  8.7  (i.e.,  /?($ljt„y„  /727))  are 
computed  using  the  same  methods  applied  in  the  EM  algorithm.  In  the  third  step,  the  0r  are 
drawn  independently  from  a multivariate  normal  distribution  with  mean  0r  and  variance  27?, 
approximating  the  distribution  in  Equation  8.7.  These  three  steps  are  repeated  five  times 
producing  five  imputations  or  plausible  values  of  0r  for  each  sampled  student. 


8.4  NAGB  ACHIEVEMENT  LEVELS 

Since  its  beginning,  a goal  of  N AEP  has  been  to  inform  the  public  about  what  students  in 
American  schools  know  and  can  do.  While  the  NAEP  scales  provide  information  about  the 
distributions  of  proficiency,  estimated  by  scale  scores,  for  the  various  subpopulations,  they  do 
not  directly  provide  information  about  the  meaning  of  various  points  on  the  score  scale. 
Traditionally,  meaning  has  been  attached  to  educational  scales  by  norm-referencing — that  is,  by 
comparing  students  at  a particular  scale  level  to  other  students.  Beginning  in  1990,  NAEP  reports 
have  also  presented  data  using  achievement  levels.  The  reading  achievement  levels  were 
developed  a. id  adopted  by  the  National  Assessment  Governing  Board  (NAGB),  as  authorized  by 
the  NAEP  legislation.  The  achievement  levels  describe  selected  points  on  the  scale  in  terms  of 
the  types  of  skills  that  are  or  should  be  exhibited  by  students  scoring  at  that  level.  The 
achievement  level  process  was  applied  to  the  1992  national  NAEP  mathematics  composite  and 
the  1994  national  scales  were  linked  to  the  1992  national  scales.  Since  the  Trial  State  Assessment 
scales  were  linked  to  the  national  scales  in  both  years,  the  interpretations  of  the  selected  levels 
also  apply  to  the  State  Assessments  in  1990,  1992,  and  1996. 

NAGB  has  determined  that  achievement  levels  shall  be  the  first  and  primary  way  of 
reporting  NAEP  results.  Setting  achievement  levels  is  a method  for  setting  standards  on  the 
NAEP  assessment  that  identify  what  students  should  know  and  be  able  to  do  at  various  points  on 
the  reading  composite.  For  each  grade  in  the  national  assessment  and,  here,  for  grades  4 and  8 in 
the  State  Assessment,  four  levels  were  defined — basic,  proficient,  advanced , and  the  region 
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below  basic.  Based  on  initial  policy  definitions  of  these  levels,  panelists  were  asked  to  determine 
operational  descriptions  of  the  levels  appropriate  with  the  content  and  skills  assessed  in  the 
mathematics  assessment.  With  these  descriptions  in  mind,  the  panelists  were  then  asked  to  rate 
the  assessment  items  in  terms  of  the  expected  performance  of  marginally  acceptable  students  at 
each  of  these  levels.  These  ratings  were  then  mapped  onto  the  NAEP  scale  to  obtain  the 
achievement  level  cutpoints  for  reporting.  Further  details  of  the  achievement  level-setting 
process  appear  in  Appendix  F. 


8.5  ANALYSES 

When  survey  variables  are  observed  without  error  from  every  student,  standard  variance 
estimators  quantify  the  uncertainty  associated  with  sample  statistics  from  the  only  source  of 
uncertainty,  namely  the  sampling  of  students.  Item-level  statistics  for  NAEP  cognitive  items  meet 
this  requirement,  but  scale-score  values,  which  estimate  proficiency,  do  not.  The  IRT  models 
used  in  their  construction  posit  an  unobservable  proficiency  variable  0to  summarize 
performance  on  the  items  in  the  subarea.  The  fact  that  lvalues  are  not  observed  even  for  the 
students  in  the  sample  requires  additional  statistical  analyses  to  draw  inferences  about  6 
distributions  and  to  quantify  the  uncertainty  associated  with  those  inferences.  As  described 
above,  Rubin’s  (1987)  multiple  imputations  procedures  were  adapted  to  the  context  of  latent 
variable  models  to  produce  the  plausible  values  upon  which  many  analyses  of  the  data  from  the 
State  Assessment  were  based.  This  section  describes  how  plausible  values  were  employed  in 
subsequent  analyses  to  yield  inferences  about  population  and  subpopulation  distributions  of  scale 
score  averages. 


8.5.1  Computational  Procedures 

Even  though  one  does  not  observe  the  lvalue  of  student  /,  one  does  observe  variables 
that  are  related  to  it:  Xj,  the  student’s  answers  to  the  cognitive  items  he  or  she  was  administered  in 
the  area  of  interest,  and  the  student’s  answers  to  demographic  and  background  variables. 
Suppose  one  wishes  to  draw  inferences  about  a number  T(d,Y)  that  could  be  calculated  explicitly 
if  the  dand  y values  of  each  member  of  the  population  were  known.  Suppose  further  that  if  6 
values  were  observable,  we  would  be  able  to  estimate  T from  a sample  of  N pairs  of  0 and  \ 

values  by  the  statistic  t(0,y)  [where  (6,y)  = (6hy, <2v.yw)L  and  that  we  could  estimate  the 

variance  in  t around  Tdue  to  sampling  students  by  the  function  U(0,yj.  Given  that  observations 
consist  of  (&.%)  rather  than  (&£),  we  can  approximate  t by  its  expected  value  conditional  on 
(.££),  or 


t‘ (*,yj  = E[t(0,y)\x,yj  = ft(0y)p(0  d§- 

It  is  possible  to  approximate  /*  with  random  draws  from  the  conditional  distributions 
p(6,\x,.yj),  which  are  obtained  for  all  students  by  the  method  described  in  Section  8.3.3.  Let  6,„  be 
the  mth  such  vector  of  plausible  values,  consisting  of  a multidimensional  value  for  the  latent 
variable  of  each  student.  This  vector  is  a plausible  representation  of  what  the  true  (?  vector  might 
have  been,  had  we  been  able  to  observe  it. 
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The  following  steps  describe  how  an  estimate  of  a scalar  statistic  t(d,y)  and  its  sampling 
variance  can  be  obtained  from  M (>1)  such  sets  of  plausible  values.  (Five  sets  of  plausible  values 
are  used  in  analyses  of  the  NAEP  State  Assessment.) 


1 .  Using  each  set  of  plausible  values  dm  in  turn,  evaluate  t as  if  the  plausible  values 
were  true  values  of  6.  Denote  the  results  tm  for  m=l,...,M. 


2.  Using  the  jackknife  variance  estimator  defined  in  Chapter  7,  compute  the  estimated 
sampling  variance  of  tm,  denoting  the  result  Um. 


3.  The  final  estimate  of  t is 


* 

t 


M 

■ i 


A 

t m 

M ' 


Compute  the  average  sampling  variance  over  the  M sets  of  plausible  values,  to 
approximate  uncertainty  due  to  sampling  students: 

" Um 

M 


m 


m-l 


5. 


Compute  the  variance  among  the  M estimates  tm,  to  approximate  uncertainty  due  to 
not  observing  lvalues  from  students: 


B = 


M /-  « ,2 

y iLw—lJJ. 
(M  - 1) 


6.  The  final  estimate  of  the  variance  of  /*  is  the  sum  of  two  components: 


V — U + (I  + M1)  B. 


Note:  Due  to  the  excessive  computation  that  would  be  required,  NAEP  analyses  did  not 
compute  and  average  jackknife  variances  over  all  five  sets  of  plausible  values,  but  only  on  the 
first  set.  Thus,  in  NAEP  reports,  if  is  approximated  by  U /. 


8.5.2  Statistical  Tests 

Suppose  that  if  lvalues  were  observed  for  sampled  students,  the  statistic  (t  - T)/Uw 
would  follow  a /-distribution  with  d degrees  of  freedom.  Then  the  incomplete-data  statistic 
(t  - 7yV,/?  is  approximately  /-distributed,  with  degrees  of  freedom  given  by: 

/ 

f a -ft 

M-l  d 

where /is  the  proportion  of  total  variance  due  to  not  observing  lvalues: 

/=  (l+M’)B/V. 
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When  B is  small  relative  to  U\  the  reference  distribution  for  incomplete-data  statistics 
differs  little  from  the  reference  distribution  for  the  corresponding  complete-data  statistics.  This  is 
the  case  with  main  NAEP  reporting  variables.  If,  in  addition,  d is  large,  the  normal 
approximation  can  be  used  to  flag  “significant”  results. 

For  Jfc-dimcnsional  r,  such  as  the  k coefficients  in  a multiple  regression  analysis,  each  Um 
and  U'  is  a covariance  matrix,  and  B is  an  average  of  squares  and  cross-products  rather  than 
simply  an  average  of  squares.  In  this  case,  the  quantity  (T-t‘)  V'  ( T-t  )’  is  approximately  F 
distributed,  with  degrees  of  freedom  equal  to  k and  v,  with  v defined  as  above  but  with  a matrix 
generalization  of /: 


/ =(1+M  ')  Trace  (BV’)/k. 

By  the  same  reasoning  as  used  for  the  normal  approximation  for  scalar  t,  a chi-square  distribution 
on  k degrees  of  freedom  often  suffices. 


8.5.3  Biases  in  Secondary  Analyses 

Statistics  t‘  that  involve  scale  score  averages  in  a scaled  content  strand  and  variables 
included  in  the  conditioning  variables  y are  consistent  estimates  of  the  corresponding  population 
values  T.  Statistics  involving  background  variables  y that  were  not  conditioned  on,  or 
relationships  among  scale  score  averages  from  different  content  strands,  are  subject  to  asymptotic 
biases  whose  magnitudes  depend  on  the  type  of  statistic  and  the  strength  of  the  relationships  of 
the  nonconditioned  background  variables  to  the  variables  that  were  conditioned  on  and  to  the 
score  scale  of  interest.  That  is,  the  large  sample  expectations  of  certain  sample  statistics  need  not 
equal  the  true  population  parameters. 

The  direction  of  the  bias  is  typically  to  underestimate  the  effect  of  nonconditioned 
variables.  For  details  and  derivations  see  Beaton  and  Johnson  (1990),  Mislevy  (1991),  and 
Mislevy  and  Sheehan  (1987,  Section  10.3.5).  For  a given  statistic  t'  involving  one  content  strand 
and  one  or  more  nonconditioned  background  variables,  the  magnitude  of  the  bias  is  related  to  the 
extent  to  which  observed  responses,  x,  account  for  the  latent  variable  0,  and  the  degree  to  which 
the  nonconditioned  background  variables  are  correlated  with  the  conditioning  background 
variables.  The  first  factor — conceptually  related  to  test  reliability — acts  consistently  in  that 
greater  measurement  precision  reduces  biases  in  all  secondary  analyses.  The  second  factor  acts 
to  reduce  biases  in  certain  analyses  but  increase  it  in  others.  In  particular, 

• High  shared  variance  between  conditioned  and  nonconditioned  background 
variables  mitigates  biases  in  analyses  that  involve  only  scale  score  and 
nonconditioned  variables,  such  as  marginal  means  or  regressions. 

• High  shared  variance  exacerbates  biases  in  regression  coefficients  of 
conditional  effects  for  nonconditioned  variables,  when  nonconditioned  and 
conditioned  background  variables  are  analyzed  jointly  as  in  multiple 
regression. 
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The  large  number  of  background  variables  that  have  been  included  in  the  conditioning 
vector  for  the  State  Assessment  allows  a large  number  of  secondary  analyses  to  be  carried  out 
with  little  or  no  bias,  and  mitigates  biases  in  analyses  of  the  marginal  distributions  of  d'\n 
nonconditioned  variables.  Kaplan  and  Nelson’s  analysis  of  the  1988  NAEP  reading  data  (some 
results  of  which  are  summarized  in  Mislevy,  1991),  which  had  a similar  design  and  fewer 
conditioning  variables,  indicates  that  the  potential  bias  for  nonconditioned  variables  in  multiple 
regression  analyses  is  below  10  percent,  and  biases  in  simple  regression  of  such  variables  is 
below  5 percent.  Additional  research  (summarized  in  Mislevy,  1990)  indicates  that  most  of  the 
bias  reduction  obtainable  from  conditioning  on  a large  number  of  variables  can  be  captured  by 
instead  conditioning  on  the  first  several  principal  components  of  the  matrix  of  all  original 
conditioning  variables.  This  procedure  was  adopted  for  the  State  Assessment  by  replacing  the 
conditioning  effects  by  the  first  K principal  components,  where  K was  selected  so  that  90  percent 
of  the  total  variance  of  the  full  set  of  conditioning  variables  (after  standardization)  was  captured. 
Mislevy  (1990)  shows  that  this  puts  an  upper  bound  of  10  percent  on  the  average  bias  for  all 
analyses  involving  the  original  conditioning  variables. 
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9.1  OVERVIEW 

This  chapter  describes  the  analyses  used  in  developing  the  1996  State  Assessment  mathematics 
scales.  The  procedures  used  were  similar  to  those  employed  in  the  analysis  of  the  1992  Trial  State 
Assessment  (Johnson,  Mazzeo,  & Kline,  1993)  and  are  based  on  the  philosophical  and  theoretical 
rationale  given  in  the  previous  chapter.  However,  the  nature  and  scope  of  the  1992  assessment  was 
altered  in  a number  of  ways  to  accommodate  the  evolving  nature  of  NAEP,  particularly  in  the  State 
Assessment.  The  changes  incorporated  into  the  1996  State  Assessment  included,  on  the  one  hand,  the 
assessment  of  both  public  and  nonpublic  students  for  most  jurisdictions  and  using  considerably  more 
constructed-response  items.  In  addition,  subsamples  of  the  1996  State  Assessment  were  presented  with 
different  student  inclusion  rules.  Since  comparisons  of  mathematics  results  for  1996  and  previous 
assessments  are  important,  the  sample  of  students  used  for  most  analyses  and  reporting  were  formed  so 
that  the  old  inclusion  rules  were  used.  On  the  other  hand,  estimation  items  that  were  included  in  the  1992 
Trial  State  Assessment  were  not  included  in  the  1996  State  Assessment.  Furthermore,  the  status  of  the 
assessment  has  changed.  In  1992,  the  State  Assessment  was  designated  as  a t rial , while  now  it  is 
considered  to  be  a standard  assessment  (see  Chapter  1 ). 

There  were  five  major  steps  in  the  analysis  of  the  Stale  Assessment  mathematics  data,  each  of 
which  is  described  in  a separate  section: 

• conventional  item  and  test  analyses,  and  DIF  analyses  (Section  9.3); 

• item  response  theory  (IRT)  scaling  (Section  9.4); 

• estimation  of  state  and  subgroup  proficiency  distributions  based  on  the  “plausible 
values"  methodology  (Section  9.5); 

• linking  of  the  1996  State  Assessment  scales  to  the  corresponding  scales  from  the 
1996  national  assessment  (Section  9.6);  and 

• creation  ot  the  State  Assessment  mathematics  composite  scale  (Section  9.7). 

A final  section  provides  an  explanation  of  sampling  weights  (Section  9.8). 

To  set  the  context  within  which  to  describe  the  methods  and  results  of  scaling  procedures,  a brief 
review  of  the  assessment  instruments  and  administration  procedures  is  provided. 


1 blank  Jenkins  hud  the  pi  ivii.ii>  responsibility  for  ilu*  planning  and  coordination  of  the  mathematics  stale  and  national  analyses. 
Jiahe  Qian  assisted  with  the  mathematics  state  assessment  analyses.  Computer  activities  for  these  analyses  were  directed  by 
l-dward  Kulick  and  completed  by  Steve  Wang  and  Xiaohui  Wang  Bruce  Kaplan  has  responsibility  for  preparing  student 
sampling  weights  Ini  use  m NAKI*  analyses 


9.2  DESCRIPTION  OF  ITEMS,  ASSESSMENT  BOOKLETS,  AND 
ADMINISTRATION  PROCEDURES 

The  general  design  structure  of  the  1996  State  Assessment  was  the  same  as  that  used  in  1992. 
However,  the  particulars  of  the  1996  design  differed  in  several  respects  from  those  of  1992.  First,  the 
1992  assessment  was  administered  to  public-school  students  only,  while  the  1996  assessment  included 
samples  of  both  public-  and  nonpublic-school  students.  Second,  the  1 996  assessment  used  a somewhat 
different  set  of  instruments  from  those  used  in  1992.  The  1996  item  pool  was  based  on  the  same 
curriculum  framework  used  for  1990  and  1992  national  and  Trial  State  Assessments  and  contained  five 
blocks  of  items  at  each  grade  level  that  were  identical  to  blocks  administered  in  1992.  The  1996  item 
pool,  on  the  other  hand,  included  an  expanded  number  of  blocks  containing  new  material.  The  new 
material  included  a greater  proportion  of  those  constructed-response  items.  The  longer  constructed 
response  items  were  scored  from  0 to  2,  0 to  3,  or  0 to  4.  Further,  there  were  extended  constructed- 
response  items  that  were  scored  on  a 0 to  4 scale,  requiring  about  five  minutes  to  complete.  These  item 
types  are  discussed  in  Chapter  5 with  actual  codes  used  for  the  data  files;  they  are  discussed  here  in  terms 
of  the  meaning  of  the  responses  for  analysis  purposes. 

The  fourth-grade  item  pool  contained  144  scaled  items,  each  of  which  was  classified  into  one  of 
the  five  content  strands:  59  items  for  number  sense,  properties , and  operations ; 25  items  for 
measurement ; 25  for  geometry  and  spatial  sense ; 17  for  data  analysis , statistics,  and  probability ; and  18 
for  algebra  and  functions.  These  items  consisted  of  79  multiple-choice  items,  35  constructed-response 
items  scored  dichotomously,  29  constructed-response  items  scored  polytomously,  and  1 “cluster  item.’'2 
The  items  were  divided  into  13  mutually  exclusive  blocks.  The  composition  of  each  block  of  items,  in 
terms  of  content  and  format,  is  given  in  Table  9-1 : 3 Table  9-2  shows  the  composition  of  each  block  after 
deletions  of  items  and  collapsing  of  categories  for  constructed-response  items  as  a result  of  scaling.  If 
data  had  poor  fit  with  the  response  model  for  an  item,  the  item  was  deleted.  If  a constructed  response 
item  was  scored  in  multiple  categories  but  one  category  had  no  (or  very  few)  responses  or  had  responses 
that  did  not  fit  the  data  in  some  categories,  it  was  combined  with  other  categories. 

The  eighth-grade  item  pool  contained  164  scaled  items,  54  of  which  were  common  to  the  fourth 
grade.  Each  item  was  classified  into  one  of  the  five  content  strands:  47  items  for  number  sense, 
properties,  and  operations ; 27  items  for  measurement ; 31  for  geometry  and  spatial  sense\  25  for  data 
analysis,  statistics,  and  probability ; and  34  for  algebra  and  functions.  These  items,  consisting  of  93 
multiple-choice  items,  41  constructed-response  items  scored  dichotomously,  27  constructed-response 
items  scored  polytomously,  and  3 cluster  items.  These  items  were  divided  into  13  mutually  exclusive 
blocks.  The  composition  of  each  block  of  items,  in  terms  of  content  and  format,  is  given  in  Table  9-3. 
Table  9-4  shows  the  composition  of  each  block  after  certain  items  were  dropped  or  had  a reduced  number 
of  categories  as  a result  of  scaling. 


2 A cluster  item  is  an  aggregation  of  a group  of  items  (in  the  case  of  NAEP  mathematics,  typically  three  or  four  items)  that  arc  related 
to  a single  content  strand,  topic,  or  stimulus,  and  arc  developed  and  scored  as  a single  unit  (sec  Waincr  & Kicly.  1987,  for  further 
details  and  examples  of  different  types  of  duster  items). 

vlTic  numbers  in  Tables  9d  through  9-4  differ  slightly  from  those  given  in  Chapter  2.  ‘Hie  numbers  in  Chapter  2 do  not  reflect  the 
grouping  of  certain  sets  of  items  into  cluster  items  for  the  purposes  of  sealine. 
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At  both  grades,  12  of  the  13  blocks  contained  one  or  more  constructed-response  items.  Six  blocks 
at  grade  4 and  eight  blocks  at  grade  8 contained  extended  constructed-response  items  scored  on  a to  4 
scale.  Two  blocks  at  each  grade  consisted  entirely  of  constructed-response  items.  The  items  contained  in 
one  of  the  two  constructed-response  blocks  required  the  manipulation  of  geometric  shapes  for  their 
solution.  Students  assigned  these  blocks  were  provided  a packet  containing  the  necessary  shapes  during 
the  time  period  in  which  they  worked  on  these  items.  All  constructed-response  items  were  scored  by 
specially  trained  readers,  as  described  in  Chapter  5. 

Special  materials,  including  calculators,  rulers,  and  protractors,  were  made  available  for  some 
blocks.  At  grade  4,  34  items  allowed  the  use  of  a calculator  for  their  solutions.  These  items  appeared  in 
three  of  the  blocks  (15  items  in  block  M8, 9 in  Ml 2,  and  10  in  Ml 4).  At  grade  8,  47  items  that  allowed 
the  use  of  a calculator  appeared  in  three  blocks  (18  items  in  block  M8,  9 in  M12,  11  in  M14,  and  9 in 
block  Ml 5).  Each  student  assigned  a block  containing  items  that  allowed  the  use  of  a calculator  was 
given  a Texas  Instruments  calculator  (a  TI-108  four-function  calculator  at  grade  4,  a TI-30  scientific 
calculator  at  grade  8)  to  use  while  working  on  that  block.  For  each  item  in  a block  for  which  calculators 
were  made  available,  both  fourth-  and  eighth-grade  students  were  asked  to  indicate  whether  they  had  in 
fact  used  the  calculator  to  answer  the  item. 

Block  M15  at  grade  4 included  some  items  that  required  the  use  of  a ruler.  Block  M 15  at  grade  8 
included  some  items  that  required  the  use  of  a protractor  or  ruler.  Students  administered  these  items  were 
provided  with  the  necessary  tools  for  the  15-minute  period  during  which  they  worked  on  that  block.  Also, 
two  blocks  (M7  and  M 10)  at  both  grades  required  the  use  of  either  a package  of  manipulatives  or  a set  of 
geometric  cutouts  to  answer  certain  questions. 

At  both  grades,  the  13  blocks  were  used  to  form  26  different  booklets  according  to  a balanced 
incomplete  block  (BIB)  design  (see  Chapter  2 for  details).  Each  of  these  booklets  contained  three  blocks 
of  cognitive  items,  and  each  block  of  these  items  appeared  in  exactly  six  booklets.  To  balance  the 
possible  block  position  effects,  each  block  appeared  twice  as  the  first  block  of  items  in  a booklet,  twice 
as  the  second  block,  and  twice  as  the  third  block.  In  addition,  the  BIB  design  required  that  each  block  of 
items  be  paired  in  a booklet  with  every  other  block  of  items  exactly  once. 

The  design  of  the  1996  State  Assessment  in  mathematics  required  that  each  student  be 
administered  one  of  the  26  booklets  in  the  BIB  design.  Within  each  administration  site,  all  booklets  were 
“spiraled”  together  in  a random  sequence  and  distributed  to  students  sequentially,  in  the  order  of  the 
students'  names  on  the  Student  Listing  Form  (see  Chapter  4).  As  a result  of  the  BIB  design  and  the 
spiraling  of  booklets,  a considerable  degree  of  balance  was  achieved  in  the  data  collection  process.  Each 
block  of  items  (and,  therefore,  each  item)  was  administered  to  randomly  equivalent  samples  of  students 
within  each  jurisdiction.  In  addition,  within  and  across  jurisdictions,  randomly  equivalent  samples 
received  each  particular  block  of  items  as  the  first,  second,  or  third  cognitive  block  within  a booklet. 

As  described  in  Chapter  4,  a randomly  selected  portion  of  the  administration  sessions  within  each 
jurisdiction  were  observed  by  Westat-trained  quality  control  monitors.  Thus,  within  and  across 
jurisdictions,  randomly  equivalent  samples  of  students  received  each  block  of  items  in  a particular 
position  within  a booklet  under  monitored  and  unmonitored  administration  conditions.  For  most 
jurisdictions  the  monitored  rate  was  about  25  percent  of  the  schools.  For  jurisdictions  that  were  new  to 
the  state  assessments  (Alaska,  Department  of  Defense  Domestic  Dependent  Elementary  and  Secondary 
Schools  (DDESS),  Nevada,  Vermont,  and  Washington)  50  percent  of  the  sessions  were  monitored.  A 
comparison  of  the  item  statistics  under  monitored  and  unmonitored  conditions  is  made  in  the  tables 
included  in  the  next  section.  Other  results  of  monitoring  are  described  in  Chapter  4. 
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Grade  4 (As  defined  before  scaling) 
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Table  9-2 

1996  NAEP  Mathematics  Block  Composition  by  Content  Strand  and  Item  Type * 
Grade  4 (As  defined  after  scaling.  Counts  reflect  items  that  were  dropped  and  collapsed) 
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Table  9-3 

1996  NAEP  Mathematics  Block  Composition  by  Content  Strand  and  Item  Type 
Grade  8 (As  defined  before  scaling) 
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Table  9-4 

1996  NAEP  Mathematics  Block  Composition  by  Content  Strand  and  Item  Type* 
Grade  8 (As  defined  after  scaling.  Counts  reflect  items  that  were  dropped  and  collapsed) 
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9.3 


ITEM  ANALYSES 


9.3.1  Conventional  Item  and  Test  Analyses 

Tables  9-5  through  9-8  contain  summary  statistics  for  each  block  of  items  for  the  fourth  and  the 
eighth  grades,  respectively.  These  statistics  summarize  characteristics  of  the  items  as  they  were  treated  in 
scaling.  (See  Table  9-22  for  special  treatment  of  some  items.)  The  tables  provide  the  item  statistics 
separately  for  public-  and  nonpublic-school  students.  Block-level  statistics  are  provided  both  overall  and 
by  serial  position  of  the  block  within  booklet.  To  produce  these  tables,  data  from  all  47  jurisdictions  were 
aggregated  and  statistics  were  calculated  using  rescaled  versions  of  the  final  reporting  weights  provided 
by  Westat.  The  rescaling,  carried  out  within  each  jurisdiction,  constrained  the  sutn  ot  the  sampling 
weights  for  public-school  students  within  that  jurisdiction  to  be  equal  to  2,000. 1 he  same  rescaling  {actor 
was  then  applied  to  the  weights  of  the  nonpublic-school  students.  Use  ot  the  rescaled  weights  does 
nothing  to  alter  the  value  of  statistics  calculated  separately  within  each  jurisdiction.  However,  for 
statistics  obtained  from  samples  that  combine  students  from  different  jurisdictions,  use  of  the  rescaled 
weights  results  in  a roughly  equal  contribution  of  each  jurisdiction's  data  to  the  final  value  ot  the 
estimate.  As  discussed  in  Mazzeo  (1991),  equal  contribution  of  each  jurisdiction’s  data  to  the  results  ot 
the  IRT  scaling  was  viewed  as  a desirable  outcome  and,  as  described  in  the  scaling  section  below, 
similarly  rescaled  weights  were  used  in  carrying  out  that  scaling.  Hence,  the  item  analysis  statistics 
shown  in  Tables  9-5  and  9-6  are  consistent  with  the  weighting  used  in  scaling.  Section  9.8  contains  more 
detailed  information  about  the  weights  used  in  the  1996  State  Assessment  in  mathematics. 


Specifically,  Tables  9-5  through  9-8  show  the  number  of  students  assigned  each  block  of  items, 
the  average  item  score,  the  average  biserial  correlation,  and  the  proportion  of  students  attempting  the  last 
item  in  that  block.  The  average  item  score  for  the  block  is  the  average,  over  items,  of  the  score  means  lor 
each  of  the  individual  items  in  the  block.  For  dichotomously-scored  multiple-choice  and  constructed- 
response  items  (0,1  scoring),  these  score  means  correspond  to  the  proportion  of  students  who  correctly 
answered  each  item.  For  the  cluster  items  and  more  than  2-category  constructed-response  items,  the  score 
means  were  calculated  as  item  score  mean  divided  by  the  maximum  number  of  points  possible. 


In  NAEP  analyses  (both  conventional  and  IRT-based),  a distinction  is  made  between  missing 
responses  at  the  end  of  each  block  (i.e..  missing  responses  subsequent  to  the  last  item  the  student 
answered)  and  missing  responses  prior  to  the  last  observed  response.  Missing  responses  before  the  last 
observed  response  are  considered  intentional  omissions.  In  calculating  the  average  score  lor  each  item, 
only  students  classified  as  having  been  presented  the  item  were  included  in  the  denominator  of  the 
statistic.  Intentional  omissions  were  considered  “omitted"  and  were  treated  as  incorrect  responses. 
Missing  responses  at  the  end  of  the  block  arc  considered  “not-reached,"  and  treated  as  it  they  had  not 
been  presented  to  the  student.  The  proportion  of  students  attempting  the  last  item  of  a block  (or, 
equivalently.  1 minus  the  proportion  of  students  not  reaching  the  last  item)  is  often  used  as  an  index  of 
the  degree  of  speededness  associated  with  the  administration  of  that  block  of  items. 

Prior  to  1992,  standard  practice  at  FITS  was  to  treat  all  nonrespondents  to  the  last  item  as  if  they 
had  not  reached  the  item.  For  multiple-choice  and  constructed-response  items  that  only  required  short 
answers  and  are  scored  on  a 0, 1 , a 0 to  2.  or  a 0 to  3 scale,  the  use  of  such  a convention  most  often 
produces  a reasonable  pattern  of  results  in  that  the  proportion  reaching  the  last  Hem  is  not  dramatically 
smaller  than  the  proportion  reaching  the  ncxt-to-last  item.  However,  for  the  blocks  that  ended  with  extended 
constructed-response  items  that  require  answers  taking  up  to  five  minutes  to  produce,  use  ot  the  standard 
ETS  convention  resulted  in  an  extremely  large  drop  in  the  pioportion  of  students  attempting  the  final  item. 
A drop  of  such  magnitude  seemed  somewhat  implausible  under  the  assumption  that  all  nonresponse  is  due 
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to  not  reaching  the  item  rather  than  intentionally  deleting  the  item.  Therefore,  for  blocks  ending  with  an 
extended  constructed-response  item,  students  who  answered  the  next-to-last  item  but  did  not  respond  to  the 
extended  constructed-response  item  were  classified  as  having  intentionally  omitted  the  last  item. 

Dichotomously-scored  items  were  analyzed  using  standard  procedures  that  result  in  a report  for 
each  item  that  includes: 

• for  each  option  of  the  item,  for  examinees  omitting  and  not  reaching  the  item,  and  for 
the  total  sample  of  examinees: 

=>  number  of  examinees, 

=>  percentage  of  examinees, 

=>  mean  of  number-correct  scores,  and 
=>  standard  deviation  of  number-correct  scores; 

• p+,  the  proportion  of  examinees  that  received  a correct  score  on  the  item  (ratio  of 
number  correct  to  number  correct  plus  wrong  plus  omitted); 

• A,  the  inverse-normally  transformed  p+  scaled  to  mean  1 3 and  standard  deviation  4; 

• the  biserial  correlation  coefficient  between  the  item  and  the  number-correct  scores; 
and 

• the  point-biserial  correlation  coefficient  between  the  item  and  the  number-correct 
scores. 

Polytomously-scored  items  were  analyzed  using  the  following  procedures: 

• in  place  of  p+,  the  ratio  of  the  mean  item  score  to  the  maximum  possible  item  score 
was  used; 

• in  place  of  A,  the  ratio  of  the  mean  item  score  to  the  maximum  possible  item  score 
underwent  the  same  transformation  as  that  used  on  p+  to  get  A for  dichotomously 
scored  items; 

• the  polyserial  correlation  coefficient  was  used  in  place  of  the  biserial;  and 

• the  Pearson  correlation  coefficient  was  used  in  place  of  the  point-biserial. 

The  average  biserial  correlation  is  the  average,  over  items,  of  the  item-level  polyserial  correlations 
(r-biserial)  between  the  item  and  the  number  correct  block  score.  For  each  item-level  r-biserial,  total  block 
number-correct  score  (including  the  item  in  question,  and  with  students  receiving  zero  points  for  all  not- 
reached  items)  was  used  as  the  criterion  variable  for  the  correlation.  The  number  correct  score  was  the  sum 
of  the  item  scores  where  correct  dichotomous  items  are  assigned  1 and  correct  polytomous  (or  multiple- 
category)  items  are  assigned  the  score  category  assigned  to  the  student.  Data  from  students  classified  as  not 
reaching  the  item  were  omitted  from  the  calculation  of  the  statistic. 


As  is  evident  from  Tables  9-5  through  9-8,  the  difficulty  and  the  average  item-to-total  correlations 
of  the  blocks  varied  somewhat.  Such  variability  was  expected  since  these  blocks  were  not  created  to  be 
parallel  in  either  difficulty  or  content.  Based  on  the  proportion  of  students  attempting  the  last  item  for 
public-school  students,  only  one  block  for  the  fourth  grade  and  one  block  for  the  eighth  grade  seemed  to  be 
somewhat  speeded,  in  that  they  had  proportions  less  than  .8.  Only  69  percent  of  the  fourth-grade  students 
taking  block  M 14  and  68  percent  of  the  eighth-grade  students  taking  block  M8  reached  the  last  item  in  the 
block! 


These  tables  also  indicate  that  there  was  little  variability  in  average  item  scores  or  average 
polyserial  correlations  for  each  block  by  serial  position  within  the  assessment  booklet.  This  suggests  that 
serial  position  within  booklet  had  a negligible  effect  on  the  overall  difficulty  of  the  block.  However,  for  the 
fourth  grade,  one  aspect  of  block  level  performance  that  did  differ  by  serial  position  was  the  proportion  of 
students  attempting  the  last  item  in  the  block.  As  shown  in  Tables  9-5  and  9-6,  the  majority  of  the  blocks 
showed  increases  in  the  proportion  attempting  the  last  item  when  moving  from  the  first  to  third  position. 
Perhaps  fourth-grade  students  are  able  to  work  more  quickly  in  later  blocks  or  are  better  able  to  pace 
themselves  as  a result  of  their  experience  with  the  first  block  of  items  that  they  attempt.  It  is  interesting  to 
note  that  this  effect  was  particularly  salient  for  blocks  M8,  M10,  Ml  1,  M12,  and  M14  in  the  public-school 
sector  (the  items  in  these  blocks  required  the  manipulation  of  geometric  shapes  and  allowed  the  use  of 
calculators  for  their  solution).  Only  two  blocks  at  grade  8 showed  a substantial  position  effect,  M8  and 
M 1 5 (the  items  in  these  blocks  allowed  the  use  of  calculators  for  their  solution).  Note  that  these  blocks  do 
not  have  a greater  proportion  of  polytomous  items.  Tables  9-2  and  9-4  indicate  that  the  blocks  with  position 
effects  had  items  of  similar  composition  as  the  other  blocks  in  the  assessment. 

As  mentioned  earlier,  to  maintain  rigorous  standardized  administration  procedures  across  the 
jurisdictions,  a randomly  selected  25  percent  of  all  sessions  within  each  trend  jurisdiction  (50%  for  new 
jurisdictions)  were  observed  by  a Westat-trained  quality  control  monitor.  The  monitoring  was  done  in 
similar  proportions  for  the  public  and  nonpublic  schools.  Observations  from  this  random  portion  of  the 
sessions  provided  information  about  the  quality  ot  administration  procedures  and  the  frequency  of 
departures  from  standardized  procedures  in  the  monitored  sessions  (see  the  last  section  in  Chapter  4 for  a 
discussion  of  the  results  of  these  observations).  Unexpectedly  large  differences  in  results  from  monitored 
and  unmonitored  sessions  (i.e.,  differences  larger  than  those  to  be  expected  due  to  sampling  fluctuation) 
could  be  used  to  identify  instances  of  cheating,  breaches  of  test  security,  or  other  breaks  in  standardization 
occurring  in  the  unmonitored  sessions  that  might  threaten  the  validity  of  assessment  results. 

When  results  were  aggregated  over  all  participating  jurisdictions,  there  was  little  difference 
between  the  performance  of  students  who  attended  monitored  or  unmonitored  sessions.  The  average  item 
score  (over  all  13  blocks  and  over  all  47  participating  jurisdictions)  for  all  fourth-grade  students  in  the  State 
Assessment  was  .54  (including  monitored  plus  unmonitored  sessions  and  public  plus  nonpublic  schools). 
The  average  item  score  for  all  eighth-grade  students  was  .57.  Tables  9-9  through  9-12  provide,  for  each 
block  of  items,  the  average  item  score,  average  polyserial,  and  the  proportion  of  students  attempting  the  last 
item  for  students  whose  sessions  were  monitored  and  students  whose  sessions  were  unmonitored,  in  public 
and  nonpublic  schools.  For  both  grades,  little  or  no  differences  in  average  item  performance  by  session  type 
were  evident.  This  result  held  up  for  both  public  and  nonpublic  schools.  The  results  for  the  public  schools 
arc  quite  consistent  with  those  observed  in  the  1990  and  1992  Trial  State  Assessments,  where  no  evidence 
was  found  that  students  who  attended  monitored  sessions  performed  differently  than  those  who  attended 
unmonitored  sessions. 

Table  9-13  summarizes  the  differences  between  monitored  and  unmonitored  average  item  scores 
for  the  jurisdictions.  These  tire  mean  differences  within  a jurisdiction  averaged  over  all  items  in  all  the 
blocks.  The  information  in  this  table  combines  public-  and  nonpublic-school  data. 
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At  the  fourth  grade,  the  median  difference  (monitored  minus  unmonitored)  was  -.000,  almost 
zero.  For  24  jurisdictions,  the  difference  was  negative  (i.e.,  students  from  unmonitored  sessions  scored 
higher  than  students  from  monitored  sessions),  with  the  largest  difference  being  -.041 . For  the  remaining 
23  jurisdictions,  the  difference  was  positive,  with  the  largest  difference  being  .039.  In  evaluating  the 
magnitude  of  these  differences,  it  should  be  noted  that  the  standard  error  for  a difference  in  proportions 
from  independent  simple  random  samples  of  size  625  (one  quarter  of  the  typical  total  jurisdiction  sample 
size  of  2,500)  from  a population  with  a true  proportion  of  .5  is  about  .028.  For  samples  with  complex 
sampling  designs  like  NAEP,  the  standard  errors  tend  to  be  larger  than  those  associated  with  simple 
random  sampling.  A reasonable  estimate  of  the  design  effect  for  average  item  scores  based  on  past  NAEP 
experience  with  item  proportion  correct  statistics  is  about  1.5  (Johnson  & Rust,  1992),  which  suggests 
that  a typical  estimate  of  the  standard  error  of  the  difference  between  monitored  and  unmonitored 
sessions  would  be  about  .03.  Only  four  of  the  47  participants  had  absolute  differences  larger  than  or 
equal  to  .03.  In  summary,  differences  in  results  obtained  from  the  two  types  of  sessions  at  the  fourth 
grade  were  well  within  the  bounds  expected  due  to  sampling  fluctuation. 

At  the  eighth  grade,  the  median  difference  was  essentially  zero  (.005).  For  18  jurisdictions,  the 
differences  were  negative.  Five  were  larger  in  absolute  magnitude  than  .03,  all  of  which  were  positive. 
The  results  indicate  that  across  jurisdictions,  there  were  only  small  differences  between  monitored  and 
unmonitored  sessions. 
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Table  9-5 

Descriptive  Statistics  for  Each  Block  of  Items  by  Position  Within  Test  Booklet  and  Overall 
Public-School  Sessions,  Grade  4 (As  defined  after  scaling) 
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Table  9-6 

Descriptive  Statistics  for  Each  Block  of  Items  by  Position  Within  Test  Booklet  and  Overall 
Nonpublic-School  Sessions,  Grade  4 (A  s defined  after  scaling) 
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Table  9-7 

Descriptive  Statistics  for  Each  Block  of  l tetris  by  Position  Within  Test  Booklet  and  Overall 
Public-School  Sessions,  Grade  8 (As  defined  after  scaling) 
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Table  9-8 

Descriptive  Statistics  for  Each  Block  of  Items  by  Position  Within  Test  Booklet  and  Overall 
Nonpublic-School  Sessions,  Grade  8 (As  defined  after  scaling) 
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Tab,e  99 

Block-Level  Descriptive  Statistics  for  Monitored  and  Unmonitored  Public-School  Sessions 

Grade  4 (As  defined  after  scaling) 
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Table  9-10 

Block-Level  Descriptive  Statistics  for  Monitored  and  Unmonitored  Nonpublic-School  Sessions 

Grade  4 (As  defined  after  scaling) 
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Grade  8 (As  defined  after  scaling) 
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Table  9-12 

Block-L^vel  Descriptive  Statistics  for  Monitored  and  Unmonitored  Nonpublic-School  Sessions 

Grade  8 (As  defined  after  scaling) 
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Table  9-13 

The  Effect  of  Monitoring  Sessions  by  Jurisdiction: 

Average  Jurisdiction  Item  Scores  for  Monitored  and  Unmonitored  Sessions  (As  defined  after  scaling) 


Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

DoDEA/DDESS 

DoDEA/DoDDS 

District  of  Columbia 

Florida 

Georgia 

Guam 

Hawaii 

Indiana 

Iowa 

Kentucky 

Louisiana 

Maine 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

Missouri 

Montana 

Nebraska 

Nevada 

New  Hampshire 
New  Jersey 
New  Mexico 
New  York 
North  Carolina 
North  Dakota 
Oregon 
Pennsylvania 
Rhode  Island 
South  Carolina 
Tennessee 
Texas 
Utah 


Unmonitored 

0.394 

0.461 

0.432 

0.413 

0.377 

0.470 

0.505 

0.444 

0.477 

0.460 

0.324 

0.424 

0.417 

0.323 

0.410 

0.489 

0.496 

0.439 

0.380 

0.509 

0.448 

0.486 

0.470 

0.510 

0.371 

0.462 

0.475 

0.484 

0.417 

0.483 

0.391 

0.449 

0.458 

0.494 

0.468 

0.473 

0.435 

0.387 

0.430 

0.495 

0.474 


Grade  4 

Monitored 

0.400 

0.461 

0.414 

0.428 

0.428 

0.446 

0.496 

0.427 

0.446 

0.456 

0.327 

0.424 

0.399 

0.333 

0.421 

0.500 

0.467 

0.442 

0.367 

0.495 

0.443 

0.477 

0.465 

0.503 

0.387 

0.480 

0.485 

0.485 

0.432 

0.464 

0.424 

0.460 

0.443 

0.517 

0.463 

0.473 

0.455 

0.402 

0.419 

0.454 

0.472 


Monitored  - 
Unmonitored 

0.006 

0.000 

-0.018 

0.015 

0.051 

-0.024 

-0.009 

-0.017 

-0.031 

-0.004 

0.003 

0.000 

-0.018 

0.010 

0.011 

0.011 

-0.029 

0.003 

-0.013 

-0.014 

-0.005 

-0.009 

-0.005 

-0.007 

0.016 

0.018 

0.010 

0.001 

0.015 

-0.019 
0.033 
0.01 1 
-0.015 
0.023 
-0.005 
0.000 
0.020 
0.015 
-0.011 
-0.041 
-0.002 


Grade  8 

Monitored  - 

Unmonitored  Monitored  Unmonitored 


0.435 

0.519 

0.479 

0.456 

0.463 

0.510 

0.539 

0.496 

0.477 

0.513 

0.374 

0.453 

0.457 

0.386 

0.444 

0.509 

0.559 

0.470 

0.419 

0.555 


0.442 

0.536 

0.465 

0.442 

0.465 

0.519 

0.526 

0.478 

0.506 

0.508 

0.406 

0.474 

0.458 

0.463 

0.481 

0.516 

0.557 

0.476 

0.424 

0.541 


0.007 

0.017 

-0.014 

-0.014 

0.002 

0.009 

-0.013 

-0.018 

0.029 

-0.005 

0.032 

0.021 

0.001 

0.077 

0.037 

0.007 

-0.002 

0.006 

0.005 

-0.014 


0.501 

0.493 

-0.008 

0.510 

0.558 

0.048 

0.525 

0.530 

0.005 

0.563 

0.547 

-0.016 

0.394 

0.384 

-0.010 

0.509 

0.507 

-0.002 

0.547 

0.559 

0.012 

0.552 

0.552 

0.000 

0.476 

0.469 

-0.(X)7 

0.530 

0.537 

0.007 

0.527 

0.545 

0.018 

0.445 

0.481 

0.036 

0.494 

0.493 

-0.001 

0.474 

0.481 

0.007 

0.564 

0.555 

-0.009 

0.524 

0.527 

0.003 

0.492 

0.478 

-0.014 

0.453 

0.437 

-0.016 

0.457 

0.442 

-0.015 

0.490 

0.511 

0.021 

0.526 

0.503 

-0.023 
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Table  9-13  (continued) 

The  Effect  of  Monitoring  Sessions  by  Jurisdiction: 

Average  Jurisdiction  Item  Scores  for  Monitored  and  Unmonitored  Sessions  (As  defined  afier  scaling) 


Unmonitored 

Grade  4 
Monitored 

Monitored  - 

Unmonitored  Unmonitored 

Grade  8 
Monitored 

Monitored  - 
Unmonitored 

Vermont 

0.465 

0.484 

0.019 

0.529 

0.535 

0.006 

Virginia 

0.444 

0.450 

0.006 

0.483 

0.486 

0.003 

Washington 

0.458 

0.466 

0.008 

0.521 

0.527 

0.006 

West  Virginia 

0.453 

0.442 

-0.011 

0.457 

0.464 

0.007 

Wisconsin 

0.513 

0.501 

-0.012 

0.551 

0.566 

0.015 

Wyoming 

0.462 

0.441 

-0.021 

0.508 

0.526 

0.018 

mean: 

-0.001 

mean: 

0.006 

median: 

0.000 

median: 

0.005 

minimum: 

-0.041 

minimum: 

-0.023 

maximum: 

0.051 

maximum: 

0.077 

1st  quartile: 

-0.013 

1st  quartile: 

-0.008 

3rd  quartile: 

0.011 

3rd  quartile: 

0.014 

The  1996  assessment  was  the  first  time  in  a state  mathematics  assessment  that  students  from 
nonpublic  schools  were  sampled.  The  nonpublic-school  population  that  was  sampled  included  students 
from  Catholic  schools,  private  religious  schools  and  private  non-religious  schools  (all  referred  to  by  the 
term  ‘nonpublic  schools’).  Tables  9- 14  and  9-15  show  the  difference  between  public  and  nonpublic 
schools  with  respect  to  average  item  statistics,  average  r-polyserial  correlation,  and  average  proportion  of 
students  attempting  the  last  item  in  a block.  As  with  the  monitored/unmonitored  comparisons,  results 
were  aggregated  over  all  participating  jurisdictions. 


Forty-five  of  the  47  jurisdictions  that  participated  in  the  State  Assessment  had  public-school 
samples,  while  37  of  the  47  jurisdictions  had  nonpublic-school  samples. 

Consistent  differences  are  evident  between  the  public-  and  nonpublic-school  groups.  Table  9-14 
indicates  that  for  grade  4,  the  difference  in  average  item  score  between  public-  and  nonpublic-school 
students  (i.e.,  public  block  mean  minus  nonpublic  block  mean)  ranged  from  -.04  to  -.09  with  an  average 
of  -.07,  indicating  that  public-school  students  were  generally  lower  in  average  item  score.  The 
public/nonpublic  difference  in  average  item-to-total  block  correlation  (the  average  r-polyserial)  ranged 
from  .01  to  .04  with  an  average  of  .02,  indicating  that  public-school  students  generally  had  a somewhat 
higher  item-to-total  correlation.  As  for  the  proportion  of  students  attempting  the  last  item,  public  minus 
nonpublic  differences  ranged  from  +.01  to  -.04  with  an  average  ot  -.02,  indicating  that  somewhat  fewer 
students  in  public  schools  attempted  the  last  item. 

Comparisons  between  public-  and  nonpublic-school  students  were  quite  similar  for  the  eighth 
grade.  On  average,  for  public-school  students  the  average  item  score  was  lower  by  .07,  the  average  r- 
polyserial  correlation  was  greater  by  .03,  and  the  proportion  of  students  attempting  the  last  item  was 
lower  by  .02. 
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Tables  9-16  to  9-19  summarize  the  distribution  over  jurisdictions  of  the  mean  item  scores  for 
public-  and  nonpublic-school  students.  Across  jurisdictions,  the  average  public-school  scores  were  quite 
varied  with  the  difference  between  the  first  and  third  quartile  (an  indication  of  how  variable  scores  are) 
as  high  as  .07  (for  data  analysis,  statistics,  and  probability).  The  spread  of  scores  for  nonpublic-school 
students  was  very  comparable.  The  differences  between  public-  and  nonpublic-school  mean  item  scores 
was  smaller  than  the  difference  between  jurisdictions,  ranging  from  .05  {geometry  and  spatial  sense)  to 
.09  {data  analysis,  statistics,  and  probability). 

A similar  picture  emerges  for  grade  8 public-school  students  (Tables  9-18  and  9-19)  with  the 
difference  between  the  first  and  third  quartile  as  high  as  .06  (for  measurement;  geometry  and  spatial 
sense;  and  algebra  and  functions).  Nonpublic-school  students  were  about  as  variable  across  jurisdictions 
as  they  were  for  the  fourth  grade.  There  were  somewhat  greater  differences  between  the  types  of  schools, 
however,  as  great  as  . 10  for  algebra  and  functions. 
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Grade  8 (As  defined  after  scaling) 
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Table  9-16 

Distribution  of  Jurisdiction  Mean  Item  Scores  by  Content  Strand  Across  Jurisdictions 
Public  Schools,  Grade  4 (As  defined  after  scaling) 


Number  Sense,  Geometry  Data  Analysis, 


Properties,  and  and  Spatial  Statistics,  and  Algebra  and 


Operations 

Measurement 

Sense 

Probability 

Functions 

Mean 

0.468 

0.493 

0.404 

0.431 

0.375 

Median 

0.478 

0.509 

0.416 

0.444 

0.390 

1 st  Quartile 

0.445 

0.465 

0.375 

0.400 

0.349 

3rd  Quartile 

0.499 

0.525 

0.432 

0.470 

0.408 

Minimum 

0.324 

0.327 

0.272 

0.231 

0.218 

Maximum 

0.530 

0.570 

0.489 

0.539 

0.444 

Number  of 
jurisdictions 

47 

47 

47 

47 

47 

Table  9-17 

Distribution  of  Jurisdiction  Mean  Item  Scores  by  Content  Strand  Across  Jurisdictions 
Nonpublic  Schools,  Grade  4 (As  defined  after  scaling) 


Number  Sense, 
Properties,  and 
Operations 

Measurement 

Geometry 
and  Spatial 
Sense 

Data  Analysis, 
Statistics,  and 
Probability 

Algebra  and 
Functions 

Mean 

0.542 

0.569 

0.456 

0.519 

0.441 

Median 

0.552 

0.576 

0.454 

0.534 

0.448 

1st  Quartile 

0.526 

0.540 

0.431 

0.477 

0.405 

3rd  Quartile 

0.570 

0.604 

0.482 

0.554 

0.481 

Minimum 

0.465 

0.476 

0.364 

0.407 

0.336 

Maximum 

0.619 

0.678 

0.582 

0.596 

0.513 

Number  of 
jurisdictions 

36 

36 

36 

36 

36 

Table  9-18 

Distribution  of  Jurisdiction  Mean  Item  Scores  by  Content  Strand 
Public  Schools , Grade  8 (As  defined  after  scaling) 


Number  Sense, 
Properties,  and 
Operations 

Measurement 

Geometry 
and  Spatial 
Sense 

Data  Analysis, 
Statistics,  and 
Probability 

Algebra  and 
Functions 

Mean 

0.499 

0.400 

0.514 

0.470 

0.542 

Median 

0.499 

0.398 

0.515 

0.480 

0.543 

1st  Quartile 

0.472 

0.372 

0.484 

0.443 

0.508 

3rd  Quartile 

0.531 

0.439 

0.552 

0.503 

0.581 

Minimum 

0.359 

0.250 

0.346 

0.334 

0.389 

Maximum 

0.57 1 

0.477 

0.590 

0.533 

0.618 

Number  of 
jurisdictions 

47 

47 

47 

47 

47 

Table  9-19 

Distribution  of  Jurisdiction  Mean  Item  Scores  by  Content  Strand 
Nonpublic  Schools , Grade  8 (As  defined  after  scaling) 

Number  Sense, 
Properties,  and 
Operations 

Measurement 

Geometry 
and  Spatial 
Sense 

Data  Analysis, 
Statistics,  and 
Probability 

Algebra  and 
Functions 

Mean 

0.586 

0.473 

0.592 

0.544 

0.640 

Median 

0.582 

0.470 

0.594 

0.543 

0.639 

1st  Quartile 

0.564 

0.437 

0.560 

0.517 

0.607 

3rd  Quartile 

0.609 

0.500 

0.621 

0.566 

0.669 

Minimum 

0.526 

0.406 

0.510 

0.482 

0.563 

Maximum 

0.656 

0.569 

0.670 

0.612 

0.728 

Number  of 
jurisdictions 

33 

33 

33 

33 

33 

9.3.2  Differential  Item  Functioning  (DIF)  Analyses 

Before  the  state  data  were  analyzed  further,  differential  item  functioning  (DIF)  analyses  were 
carried  out  on  the  1996  NAEP  mathematics  data  from  the  national  samples  at  grades  4,  8,  and  12.  The 
purpose  of  these  analyses  was  to  identify  items  that  were  differentially  difficult  for  various  subgroups 
and  to  reexamine  such  items  with  respect  to  their  fairness  and  their  appropriateness  for  inclusion  in  the 
scaling  process.  A separate  DIF  analysis  was  not  conducted  on  the  state  data  since  the  results  of  the 
national  DIF  analysis  were  assumed  to  hold  for  the  state  sample.  The  information  in  this  section  provides 
a brief  description  of  the  national  DIF  analysis,  though  a more  thorough  discussion  and  presentation  of 
results  based  on  the  national  assessment  will  appear  in  the  forthcoming  technical  report  for  that 
assessment. 


The  DIF  analyses  of  the  dichotomous  items  were  based  on  the  Mantel-Haenszel  chi-square 
procedure,  as  adapted  by  Holland  and  Thayer  (1988).  The  procedure  tests  the  statistical  hypothesis  that 
the  odds  of  correctly  answering  an  item  are  the  same  for  two  groups  of  examinees  that  have  been 
matched  on  some  measure  of  proficiency  (usually  referred  to  as  the  matching  criterion).  The  DIF 
analyses  of  the  polytomous  items  were  based  on  the  Mantel  (1963)  procedure  and  the  Somes  (1986)  chi- 
square  test.  These  procedures  compare  proportions  of  matched  examinees  from  each  group  in  each 
polytomous  item  response  category.  The  groups  being  compared  are  often  referred  to  as  the  focal  group 
(usually  a minority  or  other  group  of  interest,  such  as  Black  examinees  or  female  examinees)  and  the 
reference  group  (usually  White  examinees  or  male  examinees). 

For  each  dichotomous  item  in  the  assessment,  an  estimate  of  the  Mantel-Haenszel  common  odds- 
ratio,  expressed  on  the  ETS  delta  scale  for  item  difficulty  was  produced.  The  estimates  indicate  the 
difference  between  reference  group  and  focal  group  item  difficulties  (measured  in  ETS  delta  scale  units), 
and  typically  run  between  about  +3  and  -3.  Positive  values  indicate  items  that  are  differentially  easier  for 
the  focal  group  than  the  reference  group  after  making  an  adjustment  for  the  overall  mean  item  score  for 
the  two  groups.  Similarly,  negative  values  indicate  items  that  are  differentially  harder  for  the  focal  group 
than  the  reference  group.  It  is  common  practice  at  ETS  to  categorize  each  item  into  one  of  three 
categories  (Petersen,  1988):  “A”  (items  exhibiting  no  DIF),  “B”  (items  exhibiting  a weak  indication  of 
DIF),  or  “C”  (items  exhibiting  a strong  indication  of  DIF).  Items  in  category  A have  Mantel-Haenszel 
common  odds  ratios  on  the  delta  scale  that  do  not  differ  significantly  from  0 at  the  alpha  = .05  level  or 
are  less  thar.  1.0  in  absolute  value.  Category  C items  are  those  with  Mantel-Haenszel  values  that  are 
significantly  greater  than  1 and  larger  than  1 .5  in  absolute  magnitude.  Other  items  are  categorized  as  B 
items.  A plus  sign  (+)  indicates  that  items  are  differentially  easier  for  the  focal  group;  a minus  sign  (-) 
indicates  that  items  are  differentially  more  difficult  for  the  focal  group. 

The  ETS/NAEP  DIF  procedure  for  polytomous  items  incorporates  the  Mantel-Haenszel  ordinal 
procedure.  Polytomous  items  are  categorized  as  “AA,”  “BB.”  and  “CC,”  generalizations  of  the 
dichotomous  A,  B,  and  C categories. 

For  each  block  of  items  at  each  grade  a single  set  of  analyses  was  carried  out  based  on  equal- 
sized random  samples  of  data  from  all  participating  jurisdiction.  Each  set  of  analyses  involved  four 
reference  group/focal  group  comparisons:  male/female,  White/Asian  American,  White/Black,  and 
White/Hispanic.  The  first  subgroup  in  each  comparison  is  the  reference  group;  the  second  subgroup  is  the 
focal  group. 

Following  standard  practice  at  ETS  for  DIF  analyses  conducted  on  final  test  forms,  all  C and  CC 
items  were  reviewed  by  a committee  of  trained  test  developers  and  subject-matter  specialists.  Such 
committees  are  charged  with  making  judgments  about  whether  or  not  the  differential  difficulty  of  an  item 
is  unfairly  related  to  group  membership.  As  pointed  out  by  Zieky  (1993): 

It  is  important  to  realize  that  DIF  is  not  a synonym  for  bias.  The  item  response  theory  based 
methods,  as  well  as  the  Mantel-Haenszel  and  standardization  methods  of  DIF  detection, 
will  identify  questions  that  are  not  measuring  the  same  dimension(s)  as  the  bulk  of  the 
items  in  the  matching  criterion. ...Therefore,  judgement  is  required  to  determine  whether  or 
not  the  difference  in  difficulty  shown  by  a DIF  index  is  unfairly  related  to  group 
membership.  The  judgement  of  fairness  is  based  on  whether  or  not  the  difference  in 
difficulty  is  believed  to  be  related  to  the  construct  being  measured.... The  fairness  of  an  item 
depends  directly  on  the  purpose  for  which  a test  is  being  used.  For  example,  a science  item 
that  is  differentially  difficult  for  women  may  be  judged  to  be  fair  in  a test  designed  for 
certification  of  science  teachers  because  the  item  measures  a topic  that  every  entry-level 
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science  teacher  should  know.  However,  that  same  item,  with  the  same  DIF  value,  may  be 
judged  to  be  unfair  in  a test  of  general  knowledge  designed  for  all  entry-level  teachers. 

(p.  340) 

The  committee  assembled  to  review  NAEP  items  that  were  identified  as  C or  CC  items.  The 
committee  included  both  ETS  staff  and  outside  members  with  expertise  in  the  field.  It  was  the 
committee’s  judgment  that  none  of  the  C nor  CC  items  for  the  national  data  were  functioning 
differentially  due  to  factors  irrelevant  to  test  objectives.  Hence,  none  of  the  items  were  removed  from 
scaling  due  to  differential  item  functioning. 


9.4  ITEM  RESPONSE  THEORY  (IRT)  SCALING 

Items  at  each  grade  were  sorted  into  five  distinct  sets,  one  for  each  of  the  five  mathematics 
content  strands.  Table  9-20  describes  the  mean  item  scores  for  the  items  comprising  each  of  the  five 
fourth-grade  content  strands.  In  contrast  to  previous  item  mean  tables,  this  table  describes  the  item  means 
based  on  data  aggregated  across  jurisdictions,  for  each  item,  rather  than  item  means  averaged  over  items 
for  each  jurisdiction.  Table  9-21  contains  corresponding  results  for  the  eighth-grade  item  sets.  The 
averages  are  based  on  the  entire  sample  of  students  in  the  State  Assessment  and  use  the  final  reporting 
weight.  The  mean  values  indicate  that  the  fourth-grade  students  found  the  set  of  algebra  and  functions 
items  to  be  the  most  difficult.  The  eighth-grade  students  found  the  set  of  measurement  items  to  be  the 
most  difficult.  Students  at  both  grades  found  the  set  of  items  from  the  number  sense , properties , and 
operations  scale  to  be  the  easiest. 

Separate  IRT-based  scales  corresponding  to  each  of  the  item  sets  defined  above  were  developed 
using  the  scaling  models  described  in  Chapter  8.  For  each  grade,  five  scales  were  produced  by  separately 
calibrating  the  sets  of  items  classified  in  each  of  the  five  content  strands.  Since  there  were  two  grades  and 
each  had  five  scales,  a total  of  10  distinct  calibrations  were  carried  out. 

For  the  reasons  discussed  in  Mazzeo  (1991),  for  each  scale  at  each  grade,  parameters  for  each 
item  were  estimated  for  the  entire  data  set  and  used  for  all  jurisdictions.  Item  parameter  estimation  was 
carried  out  using  a 25  percent  random  sample  of  the  public-school  students  participating  in  the  1996 
State  Assessment  at  each  grade  and  included  equal  numbers  of  students  from  each  participating 
jurisdiction,  half  from  monitored  sessions  and  half  from  unmonitored  sessions.  For  the  fourth-grade 
calibrations,  the  sample  consisted  of  27,067  students,  with  602  students  being  sampled  from  each  of  the 
42  jurisdictions  (excluding  DDESS  and  DoDDS  schools)  included  in  the  scaling.4  For  the  eighth-grade 
calibrations,  the  sample  consisted  of  24,681  students,  with  548  students  being  sampled  from  the  42 
jurisdictions.  As  was  done  for  1992,  all  calibrations  were  carried  out  using  the  rescaled  sampling  weights 
described  earlier  in  an  effort  to  ensure  that  each  jurisdiction’s  data  contributed  equally  to  the 
determination  of  the  item  parameter  estimates. 

The  sample  used  for  item  calibration  was  also  constrained  to  contain  an  equal  number  of  students 
from  the  monitored  and  unmonitored  sessions  from  each  of  the  participating  jurisdictions.  To  the  extent 
that  items  may  have  functioned  differently  in  monitored  and  unmonitored  sessions,  the  single  set  of  item 
parameters  obtained  define  a set  of  item  characteristic  curves  ‘‘averaged  over”  the  two  types  of  sessions. 
Tables  9-7  and  9-8  (shown  earlier)  presented  block-level  item  statistics  that  suggested  little,  if  any, 
differences  in  item  functioning  by  session  type.  The  item  calibration  was  only  carried  out  with  the  public- 
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school  students  sampled  from  each  jurisdiction.  In  order  to  gauge  whether  items  functioned  differentially 
between  public  and  nonpublic  sessions,  a DIF  analysis  was  done  with  public-school  students  defined  as 
the  reference  group  and  nonpublic-school  students  defined  as  the  focal  group.  The  procedure  was  the 
same  as  that  for  testing  DIF  for  minority  and  gender  groups  in  the  natural  sample  (as  was  described  in 
Section  9.3  above).  Not  a single  item  was  identified  as  showing  significant  DIF  in  this  analysis  (i.e.,  there 
were  no  C nor  CC  items).  As  a result,  it  was  concluded  that  items  do  not  function  differentially  for 
public-  and  nonpublic-school  students  so  item  parameters  based  on  the  public-school  sample  were  used 

to  set  the  scales. 


Table  9-20 

Distribution  Across  Items  of  Item  Mean  Scores  Averaged  Across  All  Students  in  the  State  Assessment 

Grade  4 (As  defined  after  scaling) 


Number  Sense, 
Properties,  and 
Operations 

Measurement 

Geometry 
and  Spatial 
Sense 

Data  Analysis, 
Statistics,  and 
Probability 

Algebra  and 
Functions 

Mean: 

0.494 

0.489 

0.447 

0.480 

0.431 

Median: 

0.477 

0.422 

0.372 

0.502 

0.416 

1st  Quartile: 

0.339 

0.332 

0.289 

0.336 

0.305 

3rd  Quartile: 

0.624 

0.652 

0.605 

0.597 

0.605 

Minimum: 

0.128 

0.060 

0.066 

0.229 

0.157 

Maximum: 

0.869 

0.886 

0.918 

0.706 

0.666 

Number  of  items: 

59 

25 

25 

17 

18 

Table  9-21 

Distribution  Across  Items  of  Item  Mean  Scores  Averaged  Across  All  Students  in  the  State  Assessment 

Grade  8 (As  defined  after  scaling) 


Number  Sense,  Geometry  Data  Analysis, 

Properties,  and  and  Spatial  Statistics,  and  Algebra  and 


Operations 

Measurement 

Sense 

Probability 

Functions 

Mean: 

0.554 

0.462 

0.552 

0.496 

0.538 

Median: 

0.593 

0.441 

0.506 

0.543 

0.530 

1st  Quartile: 

0.371 

0.226 

0.365 

0.214 

0.41 1 

3rd  Quartile: 

0.717 

0.666 

0.731 

0.662 

0.657 

Minimum: 

0.107 

0.076 

0.195 

0.125 

0.242 

Maximum: 

0.915 

0.881 

0.919 

0.900 

0.947 

Number  of  items: 

47 

27 

31 

25 

34 

9.4.1  Item  Parameter  Estimation 

For  each  grade  and  each  content  strand,  item  parameter  estimates  were  obtained  by  the  NAEP 
BILOG/PARSCALF.  program,  which  combines  Mislcvy  and  Bock’s  (1982)  BILOG  and  Muraki  and 
Bock's  (1991)  PARSCALE  computer  programs.  The  program  uses  marginal  estimation  procedutes  to 
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estimate  the  parameters  of  the  one-,  two-,  and  three-parameter  logistic  models,  and  the  generalized  partial 
credit  model  described  by  Muraki  (1992)  (see  Chapter  8). 

All  multiple-choice  items  were  dichotomously  scored  (scored  0,1)  and  were  scaled  using  the 
three-parameter  logistic  model.  Omitted  responses  to  multiple-choice  items  were  treated  as  fractionally 
correct,  with  the  fraction  being  set  to  the  reciprocal  of  the  number  of  response  options  for  an  item.  All 
constructed-response  items  with  two  categories  were  dichotomously  scored  and  were  scaled  using  the 
two-parameter  logistic  model  with  the  lower  asymptote  parameter  set  at  0.  Omitted  responses  to  these 
items  were  treated  as  incorrect. 

A key  assumption  associated  with  IRT  scales  is  that  of  conditional  independence.  Conditional  on 
proficiency,  examinee's  item  responses  are  assumed  to  be  independent.  When  sets  of  items  are  logically 
dependent  on  each  other,  or  are  based  on  a single  stimulus,  this  assumption  can  be  violated  to  a degree 
that  results  in  aberrant  scaling  results.  In  order  to  avoid  possible  problems  with  inter-item  dependencies, 
several  cluster  items  (one  at  grade  4 and  three  at  grade  8)  were  created  by  combining  examinee  responses 
to  sets  of  related  items  into  a single  score  for  each  set.  The  cluster  items,  rather  than  their  original 
constituent  items,  were  used  in  scaling  the  1996  mathematics  assessment.  In  all  cases,  examinee  cluster 
item  scores  were  defined  as  the  number  of  correct  responses  given  to  each  cluster  item’s  constituent 
items.  Examinees  omitting  all  constituents  of  the  cluster  item  were  placed  in  the  “zero  correct”  category 
of  the  cluster  item.  Examinees  classified  as  “not  reaching”  all  constituent  parts  were  treated  as  having  not 
been  presented  the  cluster  item.  All  cluster  items  were  scaled  using  the  generalized  partial  credit  model. 

There  was  a total  of  56  multiple-category  constructed-response  items  at  grades  4 and  8 (29  at 
grade  4 and  27  at  grade  8).  Each  of  these  items  was  also  scaled  using  the  generalized  partial  credit  model. 
These  items  had  two,  three,  or  four  categories  of  partial  credit.  Five  scoring  levels  were  defined: 

0 Wrong,  off-task,  or  omitted 

1 Minimal  response 

2 Partially  correct 

3 Satisfactory  response 

4 Elaborated  response 

Bayes  modal  estimates  of  all  item  parameters  were  obtained  from  the  BILOG/PARSCALE 
program.  Prior  distributions  were  imposed  on  item  parameters  with  the  following  starting  values: 
thresholds  (normal  [0,2]);  slopes  (log-normal  [0,.5]);  and  asymptotes  (two-parameter  beta  with  parameter 
values  determined  as  functions  of  the  number  of  response  options  for  an  item  and  a weight  factor  of  50). 
The  locations  (but  not  the  dispersions)  were  updated  at  each  program  estimation  cycle  in  accordance  with 
provisional  estimates  of  the  item  parameters. 

As  was  done  for  the  1992  State  Assessment,  item  parameter  estimation  proceeded  in  two  phases. 
First,  the  subject  ability  distribution  was  assumed  fixed  (normal  [0,1  ])  and  a stable  solution  was  obtained. 
The  parameter  estimates  from  this  solution  were  then  used  as  starting  values  for  a subsequent  set  of  runs 
in  which  the  subject  ability  distribution  was  freed  and  estimated  concurrently  with  item  parameter 
estimates.  After  each  estimation  cycle,  the  subject  ability  distribution  was  restandardized  to  have  a mean 
of  zero  and  standard  deviation  of  one.  Correspondingly,  parameter  estimates  for  that  cycle  were  also 
linearly  restandardized. 

During  and  subsequent  to  item  parameter  estimation,  evaluations  of  the  fit  of  the  IRT  models 
were  carried  out  for  each  of  the  items  in  the  grade  4 and  grade  8 item  pools.  These  evaluations  were 
conducted  to  determine  the  final  composition  of  the  item  pool  making  up  the  scales  by  identifying 
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misfitting  items  that  should  not  be  included.  Evaluations  of  model  fit  were  based  primarily  on  graphical 
analyses.  For  dichotomously-scored  multiple-choice  and  2-category  response  items,  model  fit  was 
evaluated  by  examining  plots  of  estimates  of  the  expected  conditional  (on  0)  probability  of  a correct 
response  that  do  not  assume  a two-parameter  or  three-parameter  logistic  model  versus  the  probability 
predicted  by  the  estimated  item  characteristic  curve  (see  Mislevy  & Sheehan,  1987,  p.  302).  For  the 
cluster  items  and  multiple-category  constructed-response  items,  similar  plots  were  produced  for  each 
item  category  characteristic  curve. 

As  with  most  procedures  that  involve  evaluating  plots  of  data  versus  model  predictions,  a certain 
degree  of  subjectivity  is  involved  in  determining  the  degree  of  fit  necessary  to  justify  use  of  the  model. 
There  are  a number  of  reasons  why  evaluation  of  model  fit  relied  primarily  on  analyses  of  plots  rather 
than  seemingly  more  objective  procedures  based  on  goodness-of-fit  indices  such  as  the  “pseudo  chi- 
squares"  produced  in  BILOG  (Mislevy  & Bock.  1982).  First,  the  exact  sampling  distributions  of  these 
indie  \s  when  the  model  fits  are  not  well  understood,  even  lor  (airly  long  tests.  Mislevy  and  Stocking 
(1987)  point  out  that  the  usefulness  of  these  indices  appears  particularly  limited  in  situations  like  NAFP 
where  examinees  have  been  administered  relatively  short  tests.  A study  by  Stone,  Mislevy,  and  Mazzeo 
(1994)  using  simulated  data  suggests  that  the  correct  reference  chi-square  distributions  for  these  indices 
have  considerably  fewer  degrees  of  freedom  than  the  value  indicated  by  the  BILOG/P A.RSCALE 
program  and  require  additional  adjustments  ot  scale.  However,  it  is  not  yet  clear  how  to  estimate  the 
correct  number  of  degrees  of  freedom  and  necessary  scale  factor  adjustment  factors.  Consequently, 
pseudo  chi-square  goodness-of-fit  indices  are  used  only  as  rough  guides  in  interpreting  the  severity  of 
model  departures. 

Second,  as  discussed  in  Chapter  8.  it  is  almost  certainly  the  case  that,  for  most  items,  item- 
response  models  hold  only  to  a certain  degree  of  approximation.  Given  the  large  samples  sizes  used  in 
NAEP  and  the  State  Assessment,  there  will  be  sets  of  items  for  which  one  is  almost  certain  to  reject  the 
hypothesis  that  the  model  fits  the  data  (since  the  likelihood  of  rejecting  the  null  increases  with  sample 
size)  even  though  departures  are  minimal  in  nature  or  involve  kinds  of  misfit  unlikely  to  impact  on 
important  model-based  inferences  about  student  achievement.  In  practice,  it  is  always  wise  to  temper 
statistical  decisions  with  judgments  about  the  severity  of  model  misfit  and  the  potential  impact  of  such 
misfit  on  final  results. 

In  making  decisions  about  excluding  items  from  the  final  scales,  a balance  was  sought  between 
being  too  stringent,  hence  deleting  too  many  items  and  possibly  damaging  the  content  representativeness 
of  the  pool  of  sealed  items,  and  too  lenient,  hence  including  items  with  model  fit  poor  enough  to 
invalidate  the  types  of  inferences  made  from  NAEP  results.  Items  that  clearly  did  not  lit  the  model  were 
not  included  in  the  final  scales;  however,  a certain  degree  of  misfit  was  tolerated  for  a number  of  items 
included  in  the  final  scales. 

For  the  large  majority  of  the  grade  4 and  grade  8 items,  the  fit  of  the  model  was  extremely  good. 
Figure  9- 1 provides  a typical  example  of  what  the  plots  look  like  for  a dichotomously-scored  item  in  this 
class  of  items.  The  plot  that  is  shown  is  for  an  item  from  the  grade  8 scale.  In  the  plot,  the  y-axis 
indicates  the  probability  of  a correct  response  and  the  x-axis  indicates  scale  score  level  (0).  The  curve 
comprised  of  circles  shows  estimates  of  the  conditional  (on  0)  probability  of  a correct  response  that  do 
not  assume  a logistic  form  (referred  to  subsequently  as  nonlogistie-baxed  estimates).  The  si/es  of  the 
circles  are  proportional  to  the  estimated  density  of  the  theta  distribution  at  the  indicated  value.  The  solid 
curve  shows  the  estimated  theoretical  item  response  function.  The  item  response  function  provides 
estimates  of  the  conditional  probability  of  a correct  response  based  on  an  assumed  logistic  form.  1 he 
vertical  dashed  line  indicates  the  estimated  location  parameter  (h)  lor  the  item  and  the  horizontal  dashed 
line  indicates  the  estimated  lower  asymptote  <c).  Also  shown  in  the  plot  are  the  actual  % aloes  of  the  item 
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parameter  estimates  (lower  right-hand  corner).  As  is  evident  from  the  plot,  the  ‘empirica*  or  non- 
logistic-based  item  trace  is  in  extremely  close  agreement  with  the  model-based  item  response  function 
logistic. 


Figure  9-2  provides  an  example  of  a plot  for  a 4-category  extended  constructed-response  item 
exhibiting  good  model  fit.  Like  the  plots  for  the  dichotomously-scored  multiple-choice  items,  this  plot 
shows  two  estimates  of  each  item  category  characteristic  curve,  one  set  that  does  not  assume  the  partial 
credit  model  (the  ‘empirical’  trace  shown  as  circles)  and  one  that  does  (the  theoretical  trace  shown  as 
solid  curves).  The  dashed  horizontal  lines  show  the  location  of  the  estimated  category  thresholds  for  the 
item  (dj  to  d4;  see  Chapter  8).  The  estimates  for  all  parameters  for  the  item  in  question  are  also  indicated 
on  the  plot.  As  with  Figure  9-1,  the  two  sets  of  estimates  agree  quite  well,  although  there  is  a slight 
tendency  for  the  nonlogistic-based  estimates  for  category  two  to  be  somewhat  higher  than  the  model- 
based  estimates  for  theta  values  less  than  1 . An  aspect  of  Figure  9-2  worth  noting  is  the  large  proportion 
of  examinees  that  responded  in  the  two  lowest  response  categories  for  this  item.5  Such  results  were 
typical  for  the  extended  constructed-response  items  at  both  grades.  Substantial  proportions  of  examinees 
were  either  unable  or  unwilling  to  provide  even  minimally  adequate  answers  to  such  items. 

As  discussed  above,  some  of  the  items  retained  for  the  final  scales  display  some  degree  of  model 
misfit.  Figures  9-3  (dichotomously-scored  multiple-choice  items)  and  9-4  (extended  constructed-response 
item)  provide  typical  examples  of  such  items.  In  general,  good  agreement  between  empirical  and 
theoretical  item  traces  were  found  for  the  regions  of  the  theta  scale  that  includes  most  of  the  examinees. 
Misfit  was  confined  to  conditional  probabilities  associated  with  theta  values  in  the  tails  of  the  subject 
ability  distributions. 

Only  one  of  the  administered  items  (M07340I  in  Block  Ml 4,  a polytomous  item  measuring 
number  sense , properties,  and  operations)  was  not  included  in  the  final  scales.  The  plot  from  the  national 
scaling  for  this  item  is  given  in  Figure  9-5.  This  item  was  never  included  in  the  state  analysis.  As  is 
evident  from  the  nonlogistic-based  estimates  in  the  plots,  the  item  appears  to  have  a nonlogistic  item 
characteristic  curve.  Students  with  higher  levels  of  proficiency  exhibit  lower  chances  of  success  than  do 
students  with  lower  proficiency.  Logistic-based  estimates  of  conditional  probabilities  are,  by  definition, 
monotonically  increasing.  Hence,  the  model  does  not  fit. 


This  is  e\idonced  hy  the  relatively  large  si/c  of  the  circles  indicating  estimated  conditional  probabilities  for  these  two  categories. 


198 


231 


PROBABILITY 


Figure  9-1 

Plot*  Comparing  Empirical  and  Model-Based  Estimates  of  Item  Response  Functions 
for  a Dichotomously-Scored  Multiple-Choice  Item  Exhibiting  Good  Model  Fit 


NAEP  D M013531  SCALE  NUM&OPER 


o 


♦Circles  indicate  estimated  conditional  probabilities  obtained  without  assuming  a logistic  form:  the  solid 
curve  indicates  estimated  item  response  function  assuming  a logistic  form. 


Figure  9-2 

Plot*  Comparing  Empirical  and  Model-Based  Estimates  of  Item  Category • Characteristic  Curves 
for  a Polxtomously  Scored  hem  Exhibiting  Good  Model  Fit 


0 ft 

THETA 


* Circles  indicate  estimated  conditional  probabilities  obtained  without  assuming  a model-based  form;  the 
solid  curve  indicates  estimated  item  response  function  assuming  a model-based  form.  The  number  of 
categories  in  this  figure  includes  the  zero  category  contrary  to  usage  in  the  text  of  this  report. 
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Figure  9-3 

Plot * Comparing  Empirical  and  Model-Based  Estimates  of  Item  Response  Functions 
for  a Dichotomously-Scored  Multiple-Choice  Item  Exhibiting  Some  Model  Misfit 


NAEP  ID'  M073402  SCALE:  NUM&OPER 


o 


* Circles  indicate  estimated  conditional  probabilities  obtained  without  assuming  a logistic  form;  the  solid 
curve  indicates  estimated  item  response  function  assuming  a logistic  form. 
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Figure  9-4 

Plot * Comparing  Empirical  and  Model-Based  Estimates  of  Item  Category  Characteristic  Curves 
for  a Polytomously  Scored  Item  Exhibiting  Some  Model  Misfit 

NAEP  10.  M051101  SCALE'  NUM&OPER 


o 
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* Circles  indicate  estimated  conditional  probabilities  obtained  without  assuming  a model-based  form;  the 
solid  curve  indicates  estimated  item  response  function  assuming  a model-based  form.  The  number  of 
categories  in  this  figure  includes  the  zero  category  contrary  to  usage  in  the  text  of  this  report. 
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Figure  9-5 

Plot*  Comparing  Empirical  and  Model-Based  Estimates  of  Item  Response  Functions 
for  Items  Dropped  from  Scaling  Due  to  Model  Misfit 


■•I  0 -3  0 -2.0  -1  0 0 0 10  2 0 3 0 4 0 

THETA 

* Circles  indicate  estimated  conditional  probabilities  obtained  without  assuming  a logistic  form:  the  solid 
curve  indicates  estimated  item  response  function  assuming  a logistic  form. 
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Table  9-22  lists  the  items  that  received  special  treatment  during  the  scaling  process.  Included  in 
the  table  are  the  block  locations  and  item  numbers  for  the  items  that  were  combined  into  cluster  items  as 
well  as  for  those  that  were  excluded  from  the  final  scales.  These  items  received  identical  special 
treatment  in  the  development  of  the  1996  national  scales.  No  other  items  in  either  assessment  received 
special  treatment.  The  IRT  parameters  for  the  items  included  in  the  State  Assessment  are  listed  in 
Appendix  D. 


9.5  ESTIMATION  OF  STATE  AND  SUBGROUP  PROFICIENCY  DISTRIBUTIONS 

The  proficiency  distributions  in  each  jurisdiction  (and  for  important  subgroups  within  each 
jurisdiction)  were  estimated  by  using  the  multivariate  plausible  values  methodology  and  the 
corresponding  CGROUP  computer  program  (described  in  Chapter  8;  also  see  Mislevy,  1991).  The 
CGROUP  program  (Sheehan,  1985;  Rogers,  1991),  which  was  originally  based  on  the  procedures 
described  by  Mislevy  and  Sheehan  (1987),  was  used  in  the  1990  and  1992  Trial  State  Assessments  of 
mathematics.  The  1996  State  Assessment  used  an  enhanced  version  of  CGROUP,  based  on  modifications 
described  by  Thomas  (1993),  to  estimate  the  proficiency  distribution  for  both  the  fourth  and  the  eighth 
grades  in  each  jurisdiction.  As  described  in  Chapter  8,  the  CGROUP  program  estimates  scale  score 
distributions  using  information  from  student  item  responses,  measures  of  student  background  variables, 
and  the  item  parameter  estimates  obtained  from  the  BILOG/PARSCALE  program. 

The  enhancements  included  in  the  1992  version  of  CGROUP  included  the  replacement  of  Monte 
Carlo  integration  by  analytic  calculations,  new  methods  for  computing  student-level  posterior  means  and 
variances,  and  the  generation  of  values  from  student-level  posterior  distributions  for  the  imputation  of 
student-level  scale  score  values.  Simulation  studies  indicate  that  the  enhanced  CGROUP  produces  more 
accurate  estimates  of  scale  (i.e.  content  strand)  variances  and  correlations  (Thomas,  1993)  than  did  the 
previous  versions  of  CGROUP. 

For  the  reasons  discussed  in  Mazzeo  (1991),  separate  conditioning  models  were  estimated  at 
each  grade  for  each  jurisdiction.  This  resulted  in  the  estimation  of  47  distinct  conditioning  models  for 
each  grade.  At  each  grade,  the  student  background  variables  included  in  each  jurisdiction’s  model 
(denoted  v in  Chapter  8)  were  principal  component  scores  derived  from  the  within-state  correlation 
matrix  of  selected  main-effects  and  two-way  interactions  associated  with  a wide  range  of  student, 
teacher,  school,  and  location  variables.  The  main-effect  and  interaction  variables  are  listed  in  Appendix 
C.  A set  of  five  multivariate  plausible  values  was  drawn  for  each  student  who  participated  in  the  State 
Assessment. 

As  was  the  case  in  1992,  reporting  each  jurisdiction’s  results  required  analyses  describing  the 
relationships  between  scale  scores  and  a large  number  of  background  variables.  The  background 
variables  included  student  demographic  characteristics  (e.g.,  the  race/ethnicity  of  the  student,  highest 
level  of  education  attained  by  parents),  students’  perceptions  about  mathematics,  student  behavior  both  in 
and  out  of  school  (e.g.,  amount  of  TV  watched  daily,  amount  of  mathematics  homework  done  each  day), 
the  type  of  mathematics  class  being  taken  (e.g.,  algebra  or  general  fourth-  or  eighth-grade  mathematics), 
the  amount  of  classroom  emphasis  on  various  topics  included  in  the  assessment  provided  by  the  students’ 
teachers,  and  a variety  of  other  aspects  of  the  students’  background  and  preparation,  the  background  and 
preparation  of  their  teachers,  and  the  educational,  social,  and  financial  environment  of  the  schools  they 
attended. 
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Table  9-22 

Items  from  the  1996  State  Assessment  in  Mathematics  Receiving  Special  Treatment 


NAEP ID 

Grade 

Block 

Number  in 
Block 

Content 

Strand1 

Treatment 

M068001 

4 

M7 

5 

1 

Collapse:  (0, 1 ,2)  becomes  (0,0,1 ) 

MO  10531 

M8 

6 

1 

1992  &.  1996  responses  split  into  items 
M010531 Y and  M0I0531Z 

M074701 

M15 

6 

1 

Collapse:  (0,1,2)  becomes  (0,0,1) 

M040801 

M9 

6 

i 

1992  & 1996  responses  split  into  items 
N1040801 Y & M 04080 1 Z 

M041001 

M9 

S 

1992  & 1996  responses  split  into  items 
M04I00I Y & M041001Z 

M068003 

M7 

7 

3 

Collapse:  (0, 1 ,2)  becomes  (0,0, 1 ) 

M04120I 

M9 

10 

3 

1992  & 1996  responses  split  into  items 
M04I201  Y & M041201Z  Both  items 
collapse:  (04,2,3,4)  becomes  (0,1, 2,3, 3.) 

M072701 

M 14 

10 

5 

Collapse:  (0.1, 2, 3,4)  becomes  (0,1, 2, 3,3) 

M073401 

8 

M 14 

7 

1 

Drop 

M051 101 

M3 

13 

1 

1992  & 1996  responses  split  into  items 
M031 101 Y & M031 101 Z Both  items 
collapse:  (0,1, 2,3,4)  becomes  (0,1,2,33,) 

M069601 

Ml  2 

s 

1 

Collapse:  (0,1,2)  becomes  (0,1.1) 

M07360! 

M 14 

9 

1 

Collapse:  (0,1, 2, 3,4)  becomes  (0,1,23,3) 

MO  18201 

M4 

9 

I 

1992  & 1996  responses  split  into  items 
M01S201 Y & M0I8201Z 

MO  13531 

MR 

16 

1 

1992  & 1996  responses  split  into  items 
M0 1 353 1 Y & MO  1 353 1 Z 

M051201 

Ml  3 

I 

1 

1992  & 1996  responses  split  into  items 
M051201Y  & M051201Z 

M052201 

M 1 3 

1 1 

"> 

Collapse:  (0,1, 2,3,4)  becomes  (0. 1,233) 

M076001 

M 15 

9 

Collapse:  (0,1, 2, 3,4)  becomes  (0,1,2,33) 

M051001 

M3 

12 

3 

1992  & 1996  responses  split  into  items 
M051001Y  & M051001Z 

M068201 

M7 

10 

3 

Collapse:  (0.1, 2.3.4)  becomes  (0.0. 1,1.1) 

M05026 1 

M3 

4 

4 

Collapse:  (0,1, 2, 3,4)  becomes  (0,0, 0,1, 2) 

MO  18901 

M4 

16 

4 

1992  &.  1906  responses  split  into  items 
M0 1 890 1Y  & MO  1 890 1 Z 

M067501 

M3 

1 1 

4 

Collapse:  (0,1.23,4)  becomes  (0,1, 2, 2, 2) 

M053I01 

M9 

9 

4 

Collapse:  (04,2,3,4)  becomes  ( 0, 1 ,2,2,3 ) 

M070001 

M 12 

9 

4 

Collapse:  (04,2,3)  becomes  (0,1,23) 

M0732CL 

M 14 

3 

4 

Collapse:  (04.2,3,4)  becomes  (0,04,2,3) 

M073501 

M 14 

8 

4 

Collapse:  (0.1.2)  becomes  (0,14) 

M066301 

MS 

4 

s 

Collapse:  (0,1,2)  becomes  (04  4 ) 

M069301 

M 12 

n 

3 

Collapse:  (04,2)  becomes  (0,1,1 ) 

M0757CL 

MIS 

6 

s 

Collapse:  (04.2,3)  becomes  (0,04,2) 

'Content  strands.  1 - number  unsc,  prtywrtics.  and  oftt  raunns.  2 - tnrasttrrm*  nt.  ; - e<  nun  n\  anil  \[uinal  \cn\c:  4 data 
uniilwn.  \Uitt\tte\.  tmd  prttbuhilit\.  ill nl  S ah\t  Inn  ,tnd  tun . /m/m 
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As  described  in  the  previous  chapter,  to  avoid  biases  in  reporting  results  and  to  minimize  biases 
in  secondary  analyses,  it  is  desirable  to  incorporate  measures  of  a large  number  of  independent  variables 
in  the  conditioning  model.  When  expressed  in  terms  of  contrast-coded  main  effects  and  interactions,  the 
number  of  variables  to  be  included  totaled  870  at  grade  4 and  1 ,028  at  grade  8.  As  stated  earlier. 

Appendix  C provides  a listing  of  the  full  set  of  contrasts  defined  at  each  grade.  These  contrasts  were  the 
common  starting  point  in  the  development  of  the  conditioning  models  for  each  of  the  participating 
jurisdictions. 

Because  of  the  large  number  of  these  contrasts  and  the  fact  that,  within  each  jurisdiction,  some 
contrasts  had  zero  variance,  some  involved  relatively  small  numbers  of  individuals,  and  some  were 
highly  correlated  with  other  contrasts  or  sets  of  contrasts,  an  effort  was  made  to  reduce  the 
dimensionality  of  the  predictor  variables  in  each  jurisdiction’s  CGROUP  model.  As  was  done  for  the 
1992  Trial  State  Assessment,  the  original  background  variable  contrasts  were  standardized  and 
transformed  into  a set  of  linearly  independent  variables  by  extracting  separate  sets  of  principal 
components  (one  set  for  each  grade  for  each  of  the  47  jurisdictions)  from  the  within-state  correlation 
matrices  of  the  original  contrast  variables.  The  principal  components,  rather  than  the  original  variables, 
were  used  as  the  independent  variables  in  the  conditioning  model.  As  was  done  for  the  1992  Trial  State 
Assessment,  the  number  of  principal  components  included  for  each  jurisdiction  was  the  number  required 
to  account  for  approximately  90  percent  of  the  variance  in  the  original  contrast  variables.  Research  based 
on  data  from  the  1 990  Trial  State  Assessment  suggests  that  results  obtained  using  such  a subset  of  the 
components  will  differ  only  slightly  from  those  obtained  using  the  full  set  (Mazzeo,  Johnson,  Bowker,  & 
Fong,  1992). 

Tables  9-23  (for  grade  4)  and  9-24  (for  grade  8)  contain  a listing  of  the  number  of  principal 
components  included  in  and  the  proportion  of  scale  score  variance  accounted  for  by  the  conditioning 
model  for  each  of  the  47  participating  jurisdictions.  It  is  important  to  note  that  the  proportion  of  variance 
accounted  for  by  the  conditioning  model  differs  across  scales  within  a jurisdiction,  across  grades  within  a 
jurisdiction,  and  across  jurisdictions  within  a scale  as  summarized  in  Tables  9-23  and  9-24.  Such 
variability  is  not  unexpected  for  at  least  two  reasons.  First,  there  is  no  reason  to  expect  the  strength  of  the 
relationship  between  proficiency  and  demographics  to  be  identical  across  all  grades  and  jurisdictions.  In 
fact,  one  of  the  reasons  for  fitting  separate  conditioning  models  is  that  the  strength  and  nature  of  this 
relationship  may  differ  across  jurisdictions.  Second,  the  homogeneity  of  the  demographic  profile  also 
differs  across  jurisdictions.  As  with  any  correlational  analysis,  the  restriction  of  the  range  in  the  predictor 
variables  will  attenuate  the  relationship. 

Tables  9-25  (for  grade  4)  and  9-26  (for  grade  8)  provide  matrices  of  estimated  within-state 
correlations  among  the  five  scales  averaged  over  the  47  jurisdictions.  In  parentheses  underneath  each 
average  correlation  are  listed  the  lowest  ar.d  highest  estimated  correlation  among  the  47  jurisdictions. 

The  listed  values,  taken  directly  from  the  revised  CGROUP  program,  are  estimates  of  the  within-state 
correlations  conditional  on  the  set  oj  principal  components  included  in  the  conditioning  model. 

The  number  and  nature  of  the  scales  that  were  produced  were  consistent  with  the 
recommendations  for  reporting  that  were  given  by  the  National  Assessment  Planning  Project  (see 
Chapter  2).  Reporting  results  on  multiple  scales  is  typically  most  informative  when  each  of  the  scales 
provides  unique  information  about  the  profile  of  knowledge  and  skills  possessed  by  the  students  being 
assessed.  In  such  cases,  one  would  hope  to  see  relatively  low  correlations  among  the  scales.  However, 
with  a couple  of  exceptions,  the  correlations  among  the  1996  mathematics  scales  are  high  across  all 
jurisdictions,  almost  always  exceeding  .7  and  quite  often  exceeding  .9.  This  is  particularly  noteworthy 
w hen  one  considers  that  these  are  correlations  conditional  on  a rather  large  set  of  background  variables. 
The  marginal  correlations  between  scales  would  be  higher,  particularly  for  those  correlations  in  the  .7  to 


.8  range.  In  particular,  the  correlations  among  three  of  the  scales  — number  sense,  properties,  and 
operations',  data  analysis,  statistics,  and  probability,  and  algebra  and  functions  — are  extremely  high 
(rarely  falling  below  .9)  at  both  grades.  At  the  fourth  grade,  and  to  a somewhat  lesser  extent  at  the  eighth 
grade,  the  estimated  correlations  between  geometry  and  spatial  sense  and  the  remaining  scales  are 
noticeably  lower  than  the  correlations  among  the  remaining  scales  and  rarely  exceed  .9. 

Table  9-23 

Proportion  of  Scale  Score  Variance  Accounted  for  by  Grade  4 Conditioning  Models 


Jurisdiction 

Number 

of 

Principal 

Components 

Number 

Sense, 

Properties, 

and 

Operations 

Measurement 

Geometry 

and 

Spatial 

Sense 

Data 

Analysis, 

Statistics, 

and 

Probability 

Algebra  and 
Functions 

Alabama 

240 

0.71 

0.76 

0.62 

0.76 

0.75 

Alaska 

212 

0.72 

0.80 

0.74 

0.83 

0.83 

Arizona 

243 

0.71 

0.76 

0.66 

0.79 

0.77 

Arkansas 

208 

0.73 

0.76 

0.64 

0.81 

0.77 

California 

227 

0.74 

0.75 

0.66 

0.81 

0.76 

Colorado 

253 

0.66 

0.75 

0.59 

0.78 

0.73 

Connecticut 

267 

0.72 

0.77 

0.69 

0.80 

0.80 

Delaware 

239 

0.79 

0.84 

0.78 

0.87 

0.87 

District  Of 
Columbia 

215 

0.82 

0.87 

0.79 

0.88 

0.87 

DoDEA/ 

DDESS 

170 

0.76 

0.90 

0.83 

0.88 

(188 

DoDEA/ 

DoDDS 

237 

0.72 

0.84 

0.77 

0.84 

0.83 

Florida 

271 

0.69 

0.76 

0.63 

0.75 

0.71 

Georgia 

265 

0.69 

0.76 

0.68 

0.80 

0.74 

Guam 

175 

0.81 

0.86 

0.82 

0.87 

0.88 

Hawaii 

262 

0.75 

0.81 

0.71 

0.84 

0.85 

Indiana 

217 

0.65 

0.69 

0.60 

0.74 

0.69 

Iowa 

199 

0.63 

0.70 

0.63 

0.71 

0.70 

Kentucky 

227 

0.65 

0.71 

0.61 

0.75 

0.69 

Louisiana 

257 

0.69 

0.73 

0.67 

0.74 

0.69 

Maine 

219 

0.69 

0.75 

0.63 

0.80 

0.75 

Maryland 

227 

0.74 

0.76 

0.68 

0.79 

0.77 

Massachusetts 

235 

0.66 

0.75 

0.59 

0.72 

0.71 

Michigan 

229 

0.69 

0.73 

0.63 

0.78 

0.73 

Minnesota 

245 

0.66 

0.74 

0.59 

0.73 

0.70 

Mississippi 

255 

0.71 

0.74 

0.66 

0.79 

0.78 

Missouri 

252 

0.69 

0.75 

0.61 

0.75 

0.72 

Montana 

202 

0.72 

0.76 

0.67 

0.79 

0.75 

Nebraska 

225 

0.72 

0.77 

0.65 

0.79 

0.76 

Nevada 

239 

0.69 

0.74 

0.66 

0.78 

0.75 

New'  Jersey 

227 

0.77 

0.78 

0.69 

0.84 

0.81 

New-  Mexico 

240 

0.71 

0.75 

0.64 

0.77 

0.73 

New  York 

252 

0.73 

0.78 

0.68 

0.78 

0.72 
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Table  9-23 

Proportion  of  Scale  Score  Variance  Accounted  for  by  Grade  4 Conditioning  Models 


Jurisdiction 

Number 

of 

Principal 

Components 

Number 

Sense, 

Properties, 

and 

Operations 

Measurement 

Geometry 

and 

Spatial 

Sense 

Data 

Analysis, 

Statistics, 

and 

Probability 

Algebra  and 
Functions 

North  Carolina 

246 

0.71 

0.72 

0.63 

0.77 

0.73 

North  Dakota 

194 

0.73 

0.83 

0.65 

0.81 

0.82 

t ) iv  con 

22  2 

0.67 

0.75 

0.64 

0.76 

0.74 

lVnns\  lvnnia 

229 

0.65 

0.73 

0.57 

0.77 

0.69 

Rhode  Island 

236 

0.76 

0.84 

0.74 

0.84 

0.82 

South  Carolina 

253 

0.69 

0.73 

0.65 

0.77 

0.80 

Tennessee 

247 

0.70 

0.74 

0.65 

0.81 

0.73 

I’exas 

243 

0.60 

0.73 

0.68 

0.77 

0.76 

i tah 

231 

0.69 

0.71 

0.65 

0.75 

0.69 

Vermont 

212 

0.74 

0.79 

0.67 

0.79 

0.80 

Virginia 

261 

0.71 

0.79 

0.67 

0.79 

0.75 

Washington 

250 

0.65 

0.70 

0.59 

0.70 

0.69 

West  Virginia 

208 

0.64 

0.76 

0.62 

0.75 

0.72 

Wisconsin 

229 

0.65 

0.68 

0.54 

0.76 

0.68 

W\omimi 

213 

0.73 

0.81 

0.71 

0.82 

0.84 

Proportion  of  Scale 

Store  Variance 

Tabic  9-24 

Accounted  for 

by  Grade  S Conditioning  Models 

Jurisdiction 

Number 

of 

Principal 

Components 

Numbers, 

Properties, 

and 

Operations 

Measurement 

Geometry 

and 

Spatial 

Sense 

Data 

Analysis, 

Statistics, 

and 

Probability 

Algebra  and 
Functions 

Alabama 

270 

0.78 

0.84 

0.78 

0.82 

0.81 

Maska 

204 

0.83 

0.86 

0.79 

0.88 

0.85 

Vi/ona 

271 

0 75 

0.78 

0.73 

0.80 

0.78 

Arkansas 

245 

0.79 

0 82 

0.75 

0.82 

0.81 

( ahlotnia 

*02 

0.75 

0.82 

0.75 

0.78 

0.78 

t \ lii'i'ado 

290 

0.73 

0.80 

0.71 

0.79 

0.76 

( onneetk tit 

29S 

0.77 

0.82 

0.75 

0 78 

0.80 

1 K*ia\s  are 

235 

0.80 

0.85 

0.82 

0.83 

0.82 

1 )wtncl  ( H .. 

t ’olumbia 

205 

0.86 

0 89 

0.88 

0.89 

0.8  / 

! )oDl  A/  .. 

DDI-.SS 

1 5 2 

0.9  * 

0.96 

0.97 

(1.95 

0.95 

DoDDS 

22  * 

o.sl 

0 87 

0.81 

0.89 

0.86 

1 luridti 
( ie< m et.i 

*00 

0 74 

0 81 

071 

0.82 

0.76 

304 

0 SO 

0.81 

0 76 

0.80 

0.79 

2ns 
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Table  9-24 

Proportion  of  Scale  Score  Variance  Accounted  for  by  Grade  8 Conditioning  Models 


Jurisdiction 

Number 

of 

Principal 

Components 

Numbers, 

Properties, 

and 

Operations 

Measurement 

Geometry 

and 

Spatial 

Sense 

Data 

Analysis, 

Statistics, 

and 

Probability 

Algebra  and 
Functions 

Guam 

182 

0.87 

0.93 

0.91 

0.93 

0.90 

Hawaii 

250 

0.81 

0.84 

0.82 

0.82 

0.82 

Indiana 

264 

0.76 

0.83 

0.73 

0.82 

0.79 

Iowa 

251 

0.71 

0.75 

0.69 

0.79 

0.75 

Kentucky 

272 

0.72 

0.75 

0.66 

0.77 

0.74 

Louisiana 

294 

0.72 

0.79 

0.73 

0.78 

0.76 

Maine 

248 

0.76 

0.81 

0.77 

0.83 

0.82 

Maryland 

281 

0.80 

0.82 

0.80 

0.84 

0.82 

Massachusetts 

287 

0.73 

0.82 

0.70 

0.77 

0.76 

Michigan 

283 

0.75 

0.80 

0.72 

0.76 

0.76 

Minnesota 

287 

0.74 

0.77 

0.73 

0.80 

0.79 

Mississippi 

278 

0.73 

0.78 

0.71 

0.80 

0.78 

Missouri 

278 

0.73 

0.79 

0.71 

0.77 

0.78 

Montana 

223 

0.79 

0.81 

0.77 

0.86 

0.79 

Nebraska 

277 

0.78 

0.81 

0.73 

0.83 

0.78 

Nevada 

194 

0.81 

0.87 

0.84 

0.86 

0.86 

New  Jersey 

231 

0.76 

0.80 

0.76 

0.83 

0.81 

New  Hampshire 

265 

0.81 

0.83 

0.73 

0.80 

0.79 

New  Mexico 

272 

0.77 

0.80 

0.74 

0.83 

0.80 

New  York 

277 

0.76 

0.79 

0.72 

0.80 

0.78 

North  Carolina 

293 

0.74 

0.72 

0.72 

0.78 

0.76 

North  Dakota 

228 

0.81 

0.83 

0.68 

0.84 

0.81 

Oregon 

273 

0.77 

0.77 

0.76 

0.80 

0.81 

Rhode  Island 

251 

0.78 

0.82 

0.79 

0.84 

0.83 

South  Carolina 

280 

0.77 

0.81 

0.74 

0.83 

0.79 

Tennessee 

260 

0.76 

0.79 

0.72 

0.78 

0.78 

Texas 

292 

0.78 

0.82 

0.75 

0.81 

0.79 

Utah 

304 

0.70 

0.77 

0.68 

0.77 

0.75 

Vermont 

236 

0.85 

0.87 

0.83 

0.89 

0.84 

Virginia 

305 

0.78 

0.81 

0.74 

0.81 

0.81 

Washington 

304 

0.77 

0.83 

0.71 

0.78 

0.79 

West  Virginia 

275 

0.76 

0.80 

0.71 

0.81 

0.79 

Wisconsin 

282 

0.77 

0.79 

0.72 

0.78 

0.81 

Wyoming 

250 

0.82 

0.85 

0.77 

0.84 

0.83 

2(W 
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Table  9-25 

Average  Correlations  and  Ranges  of  Scale 
Correlations  Among  the  Mathematics  Scales  for  47  Jurisdictions,  Grade  4 


Number  Sense, 
Properties,  and 
Operations 

Measurement 

Geometry 

and 

Spatial  Sense 

Data  Analysis, 
Statistics,  and 
Probability 

Measurement 
Geometry  and 
Spatial  Sense 
Data  Analysis,  Statistics, 
and  Probability 

Algebra  and 
Functions 

.95 

(.83  to  .97) 
.76 

(.66  to  .83) 
.96 

(.87  to  .98) 
.96 

(.90to.98) 

.81 

(.71  to  .90) 
.93 

(.83  to  .97) 
.94 

(.82  to  .97) 

.79 

(.67  to  .88) 
.78 

(.68  to  .89) 

.94 

(.85  to  .98) 

Table  9-26 

A verage  Correlations 

and  Ranges  of  Scale 

Correlations 

Among  the  Mathematics 

Scales  for  47  Jurisdictions , Grade  8 

Number  Sense, 

Geometry 

Data  Analysis, 

Properties,  and 

and 

Statistics,  and 

Operations 

Measurement  Spatial  Sense 

Probability 

.96 

Measurement 

(.90  to  .99) 

Geometry  and 

.88 

.93 

Spatial  Sense 

(.76  to  .97) 

(.87  to  .96) 

Data  Analysis,  Statistics, 

.97 

.96 

.89 

and  Probability 

(.94  to  .98) 

(.90  to  .98) 

(.79  to  .95) 

Alcebra  and 

.97 

.97 

.91 

.97 

Functions 

(.94  to  .98) 

(.93  to  .98) 

(.81  to  .96) 

(.94  to  .98) 

As  discussed  in  Chapter  8.  NAHP  scales  are  viewed  as  summaries  of  consistencies  and 
regularities  that  are  present  in  item-level  data.  Such  summaries  should  agree  with  other  reasonable 
summaries  of  the  item-level  data.  In  order  to  evaluate  the  reasonableness  of  the  scale  score  results,  a 
variety  of  analyses  were  conducted  to  compare  using  scale  scoi  - 'nd  the  average  item  scores  for  each 
content  strand.  High  agreement  was  found  in  all  ot  these  analysis.  One  set  of  such  analyses  is  presented 
in  Figures  9-6  and  9-7.  The  figures  contain  scatterplots  of  the  state  public-school  item  score  mean  versus 
the  suite  public-school  scale  score  means,  for  each  of  the  five  mathematics  content  strands.  As  is  evident 
from  the  figures,  there  is  an  extremely  strong  relationship  between  the  estimates  of  state-level 
performance  in  the  scale-score  and  item-score  metrics  for  all  five  content  strands. 


Mean  Scale  Score  Mean  Scale  Score  Mean  Scale  Score 


Figure  9-6 

Plot  of  Mean  Item  Score  Versus  Mean  Scale  Score  for  Each  Jurisdiction,  Grade  4 


Number  Sense,  Properties,  Measurement 


and  Operations 


0.20  0.28  0.35  0.43  0.50  0.58  0.20  0.28  0.35  0.43  0.50  0.58 

M>3an  Item  Score  Mean  Item  Score 


Geometry  and  Spatial  Sense  Data  Analysis,  Statistics, 

and  Probability 


0.20  0.28  0.35  0.43  0.50  0.58  0.20  0.28  0.35  0.43  0.50  0.58 

Mean  Item  Score  Mean  Item  Score 

Algebra  and  Functions 


0.20  0.28  0.35  0.43  0.50  0.58 

Mean  Item  Score 


BEST  copy  AVAILABLE 
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Figure  9-7 

Plot  of  Mean  Item  Score  Versus  Mean  Scale  Score  for  Each  Jurisdiction,  Grade  8 
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9.6  LINKING  STATE  AND  NATIONAL  SCALES 


A major  purpose  of  the  State  Assessment  program  was  to  allow  each  participating  jurisdiction  to 
compare  its  1996  results  at  each  grade  level  with  the  nation  as  a whole  and  with  the  region  of  the  country 
in  which  that  jurisdiction  is  located.  Because  1996  was  the  third  round  of  the  State  Assessments,  an 
additional  goal  was  to  provide  an  opportunity  to  compare  1996  results  to  those  obtained  in  1990  and 
1992  for  those  jurisdictions  participating  in  previous  assessments. 

Although  the  fourth  and  eighth  graders  in  the  1996  State  Assessment  were  administered  the  same 
test  booklets  as  the  fourth  and  eighth  graders  in  the  national  assessment,  separate  state  and  national 
scalings  were  carried  out  (for  reasons  explained  in  Mazzeo,  1991,  and  Yamamoto  & Mazzeo,  1992). 
Again,  to  ensure  a similar  scale  unit  system  for  the  state  and  national  metrics,  the  scales  had  to  be  linked. 
Plans  for  the  scaling  of  the  1 996  national  assessment  included  procedures  linking  the  1996  scales  to  their 
1990  and  1992  counterparts.  These  procedures  were  described  in  the  technical  report  of  the  1992  national 
assessment.  Since  the  1990  and  1992  Trial  State  Assessment  scales  had  already  been  linked  to  the  1990 
and  1992  national  scales,  respectively,  linking  the  1996  State  Assessment  scales  to  their  1996  national 
counterparts  indirectly  linked  the  1996  State  Assessment  scales  to  the  1990  and  1992  Trial  State 
Assessment  scales. 

For  meaningful  comparisons  to  be  made  between  each  of  the  State  Assessment  jurisdictions  and 
the  relevant  national  samples,  results  from  these  two  assessments  had  to  be  expressed  in  terms  of  a 
similar  system  of  scale  units.  In  addition,  to  allow  for  valid  comparisons  between  grades,  the  systems  of 
scale  units  for  the  fourth-  and  eighth-grade  scales  needed  to  be  aligned  and  properly  calibrated. 
Furthermore,  the  scales  needed  to  be  comparable  to  those  used  in  1990  and  1992  to  allow  for  meaningful 
assessment  of  changes  in  proficiency  levels  for  jurisdictions  participating  in  both  assessments. 

The  fourth-  and  eighth-grade  item  pools  did  share  a set  of  common  items.  However,  as  described 
in  the  previous  section,  separate  scales  were  produced  for  the  fourth  and  eighth  grades  in  independent 
BILOG/PARSCALE  calibrations.  The  units  and  origin  of  these  scales  were  set  by  standardizing  the 
within-gradc  scale  score  distributions  for  their  respective  calibration  samples  to  have  a mean  of  zero  and 
standard  deviation  of  one.  These  arbitrary,  standard  scales  do  not  allow  for  comparisons  across  grades. 
Thus,  without  further  adjustment,  the  corresponding  grade  4 and  grade  8 scales  were  not  expressed  in 
similar  systems  of  units.  Some  form  of  scale  linking  was  required. 

The  purpose  of  this  section  is  to  describe  the  procedures  used  to  align  the  1996  State  Assessment 
scales  with  their  1996  national  counterparts.  The  procedures  that  were  used  represent  an  extension  of  the 
common  population-equating  procedures  employed  to  link  the  1990  national  and  state  scales  (Mazzeo, 

1 99 1 ; Yamamoto  & Mazzeo,  1 992). 

Using  the  house  sampling  weights  provided  by  Wesiat  (see  Section  9.8),  the  combined  sample  of 
students  from  all  participating  jurisdictions  was  used  to  estimate  the  distribution  of  scale  scores  for  the 
population  of  students  enrolled  in  public  schools  that  participated  in  the  State  Assessment Separate 
estimates  were  obtained  for  grades  4 and  8,  with  total  sample  sizes  of  108,269  and  98,724,  respectively. 
Data  from  a subsample  of  the  national  assessment  at  grades  4 and  8,  each  consisting  of  grade-eligible 
public-school  students  from  any  of  the  47  jurisdictions  that  participated  in  the  1996  State  Assessment, 
were  used  to  obtain  estimates  of  the  distribution  of  scale  scores  for  the  same  target  population.  Special 


' Students  from  ( mam,  DDI-'SS.  and  DoDDS  schools  vveie  ext  link'd  fiom  both  the  Stale  Aggregate  and  National  I .inking  samples  for 
pm  poses  ot  linking  at  grades  4 ami  S 


weights  for  this  linking  subsample  were  used.  This  subsample  of  national  data  is  referred  to  as  the 
National  Linking  sample  (NL).7  Again,  appropriate  weights  provided  by  Westat  were  used.  This  was 
done  separately  for  grades  4 and  8. 

Thus,  for  each  of  the  10  scales  (five  for  each  grade),  two  sets  of  scale  score  distributions  were 
obtained  and  used  in  the  linking  process.  One  set,  based  on  the  sample  of  combined  data  from  the  State 
Assessment  (referred  to  as  the  State  Aggregate,  or  S A,  sample)  and  using  item  parameter  estimates  and 
conditioning  results  from  that  assessment,  was  in  the  metric  of  the  1996  State  Assessment.  The  other, 
based  on  the  sample  from  the  1996  national  assessment  (NL)  and  obtained  using  item  parameters  and 
conditioning  results  from  the  national  assessment,  was  in  the  reporting  metric  of  the  1996  national 
assessment.  The  latter  metric  had  already  been  linked  to  the  1990  national  reporting  metric  using  the 
procedures  described  in  the  forthcoming  technical  report  of  the  1996  national  assessment.  The  10  State 
Assessment  and  national  scales  were  made  comparable  by  constraining  the  mean  and  standard  deviation 
of  the  two  sets  of  estimates  to  be  equal. 

More  specifically,  the  following  steps  were  followed  to  linearly  link  the  scales  of  the  two 
assessments: 

1 ) For  each  scale  at  each  grade,  estimates  of  the  scale  score  distribution  for  the  SA 
sample  was  obtained  using  the  full  set  of  plausible  values  generated  by  the  CGROUP 
program.  The  weights  used  were  the  final  sampling  weights  provided  by  Westat  (see 
Section  9.8),  not  the  rescaled  versions  discussed  in  Section  9.3.  For  each  grade  and 
each  scale,  the  arithmetic  mean  of  :he  five  sets  of  plausible  values  was  taken  as  the 
overall  estimated  mean  and  the  arithmetic  average  of  the  standard  deviations  of  the 
five  sets  of  plausible  values  was  taken  as  the  overall  estimated  standard  deviation. 

2)  For  each  scale  at  each  grade,  the  estimated  scale  score  distribution  of  the  NL  sample 
was  obtained,  again  using  the  full  set  of  plausible  values  generated  by  the  CGROUP 
program.  The  weights  used  were  specially  provided  by  Westat  to  allow  for  the 
estimation  of  scale  score  distributions  for  the  same  target  population  of  students 
estimated  by  the  jurisdiction  data.  The  means  and  standard  deviations  of  the 
distributions  (in  the  1996  national  reporting  metric)  for  each  scale  at  each  grade  were 
obtained  for  this  sample  in  the  same  manner  as  described  in  Step  1 . 

3)  For  each  scale  at  each  grade,  a set  of  linear  transformation  coefficients  was  obtained 
to  link  the  state  scale  to  the  corresponding  national  scale.  The  linking  was  of  the 
form 


Y'  = kt  + k2Y 


7 Note  that  in  1990  and  1992,  the  National  Linking  sample  was  called  the  State  Aggregate  Comparison,  or  SAC,  sample.  Many 
people  thought  this  was  easy  to  confuse  with  state  data,  so  the  term  ‘National  Linking’  or  ‘NL’  will  be  used  in  this  report. 

One  of  the  purposes  of  the  State  Assessment  was  to  allow  each  participating  jurisdiction  to  compare  its  results  with  the  nation  as 
a whole,  and  with  the  region  of  the  country  in  which  that  jurisdiction  is  located.  To  permit  such  comparisons,  nationally 
representative  samples  of  students  were  tested  as  part  of  the  national  assessment  using  the  same  assessment  booklets  as  were 
students  participating  in  the  State  Assessment.  The  national  data  to  which  the  State  Assessment  results  were  compared  came  from 
a nationally  representative  sample  of  students  in  the  fourth-  and  eighth-grade.  This  sample  was  a part  of  the  full  1996  national 
mathematics  assessment  in  which  nationally  representative  samples  of  students  in  public  and  nonpublic  schools  from  three  age 
cohorts  were  assessed:  students  who  were  in  the  fourth  grade,  eighth  grade,  or  twelfth  grade.  In  order  to  allow  for  valid 
state/nation  comparisons,  the  national  comparison  sample  of  grade-level  students  was  created  from  the  full  national  assessment 
sample  and  is  included  with  the  State  Assessment  data  files. 


where 


Y = a scale  score  level  in  terms  of  the  system  of  units  of  the 

provisional  BILOG/P ARSC ALE  scale  of  the  State  Assessment 
scaling 

L*  = a scale  score  level  in  terms  of  the  system  of  units  comparable  to 
those  used  for  reporting  the  1996  national  mathematics  results 

k2  = [Standard-DeviationNL]/[Standard-DeviationsAl 

ki  = MeanNL  - /:2[MeanSA] 

where  the  subscripts  refer  to  the  NL  sample  and  to  the  SA  sample. 

The  final  conversion  parameters  for  transforming  plausible  values  from  the  provisional 
BILOG/PARSCALE  scales  to  the  final  State  Assessment  reporting  scales  are  given  in  Table  9-27.  All 
State  Assessment  results  are  reported  in  terms  of  the  Y*  metric. 

It  is  important  to  re-emphasize  two  features  of  the  linking  procedures  just  described.  First,  the 
1996  national  scales  had  already  been  linked  to  their  1990  and  1992  counterparts.  Hence,  the  linking  just 
described  places  the  1996  state  scales  on  a metric  comparable  to  that  used  for  the  1990  and  1992  national 
scales.  Since  the  1990  and  1992  state  metric  was  also  made  comparable  to  those  same  national  scales,  the 
1990,  1992  and  1996  state  results  are  in  comparable  metrics.  Second,  the  1990  national  scales  for  each 
content  strand  were  across-grade  scales,  spanning  grades  4,  8,  and  12.  Each  had  been  produced  by 
concurrently  scaling  the  items  from  all  three  grade  levels  in  a single  BILOG  calibration  (see  Yamamoto 
& Jenkins,  1992).  For  each  content  strand,  the  grade  4 and  grade  8 1996  state  scales  have  been  linked  to 
the  same  1990  across-grade  scale.  Hence,  the  grade  4 and  grade  8 State  Assessment  results  are  also  on 
comparable  scales. 


Table  9-27 

Transformation  Constants  for  the  Grade  4 and  Grade  8 Scales 


Grade  4 

Grade  8 

Scale 

k, 

k2 

ki 

k2 

Number  Sense,  Properties,  and 
Operations 

218.69 

33.26 

273.01 

35.02 

Measurement 

224.81 

30.85 

270.38 

42.64 

Geometry  and  Spatial  Sense 

224.31 

27.81 

268.64 

30.88 

Data  Analysis,  Statistics,  and 
Probability 

223.32 

28.96 

270.74 

40.13 

Algebra  and  Functions 

225.18 

27.99 

273.19 

33.51 

As  evident  from  the  discussion  above,  a linear  method  was  used  to  link  the  scales  from  the  state 
and  national  assessments.  While  these  linear  methods  ensure  equality  of  means  and  standard  deviations 
for  the  SA  (after  transformation)  and  the  NL  samples,  they  do  not  guarantee  the  shapes  of  the  estimated 
scale  score  distributions  for  the  two  samples  to  be  the  same.  As  these  two  samples  are  both  from  a 
common  target  population,  estimates  of  the  scale  score  distribution  of  that  target  population  based  on 
each  of  the  samples  should  be  quite  similar  in  shape  in  order  to  justify  strong  claims  of  comparability  for 
the  state  and  national  scales.  Substantial  differences  in  the  shapes  of  the  two  estimated  distributions 
would  result  in  differing  estimates  of  the  percentages  of  students  above  achievement  levels  or  of 
percentile  locations  depending  on  whether  State  or  national  scales  were  used — a clearly  unacceptable 
result  given  claims  about  the  comparability  of  the  scales.  In  the  face  of  such  results,  nonlinear  linking 
methods  would  be  required. 

Analyses  were  carried  out  (one  set  of  analyses  for  grade  4 and  one  set  for  grade  8)  to  verify  the 
degree  to  which  the  linear  linking  process  described  above  produced  comparable  scales  for  State  and 
national  results.  Comparisons  were  made  between  two  estimated  scale  score  distributions,  one  based  on 
the  SA  sample  and  one  based  on  the  NL  sample,  for  each  of  the  five  mathematics  content  strand  scales. 
The  comparisons  were  carried  out  using  slightly  modified  versions  of  what  Wainer  (1974)  refers  to  as 
suspended  rootograms.  The  final  rc  orting  scales  for  the  State  and  national  assessments  were  each 
divided  into  10-point  intervals.  Two  sets  of  estimates  of  the  percentage  of  students  in  each  interval  were 
obtained,  one  based  on  the  SA  sample  and  one  based  on  the  NL  sample.  Following  Tukev  ( 1971),  the 
square  roots  of  these  estimated  percentages  were  compared. s 

The  comparisons  are  shown  in  Figures  9-8  through  9-1 2.  The  heights  ot  each  ot  the  unshaded 
bars  correspond  to  the  square  root  of  the  percentage  of  students  from  the  SA  sample  in  each  10-point 
interval  on  the  final  reporting  scale.  The  shaded  bars  show  the  differences  in  root  percents  between  the 
SA  and  NL  estimates.  Positive  differences  indicate  intervals  in  which  the  estimated  percentages  from  the 
NL  sample  are  lower  than  those  obtained  from  the  SA.  Conversely,  negative  differences  indicate 
intervals  in  which  the  estimated  percentages  from  the  NL  sample  are  higher.  For  all  five  scales  at  both 


"The  square  root  transformation  allows  tor  more  effective  comparisons  lor  counts  (or  equivalently,  percentages)  when  the  expected 
number  ol  counts  m each  interval  is  likely  to  vary  greatly  over  the, range  of  intervals,  as  is  the  case  for  the  NAFP  scales  where  the 
expected  counts  of  individuals  in  intervals  near  the  extremes  ol  the  scale  <c  g . below  ISO  and  above  ^SOl  are  dramatically  smaller 
than  the  counts  obtained  near  the  middle  ot  the  scale. 
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grades,  differences  in  root  percents  are  quite  small,  suggesting  that  the  shapes  of  the  two  estimated 
distributions  are  quite  similar  (i.e.,  unimodal  with  slight  negative  skewness).  There  is  some  evidence  that 
the  estimates  produced  using  the  SA  data  are  slightly  heavier  in  the  extreme  lower  tails  (below  100  for 
the  grade  4 scales  and  below  150  for  the  grade  8 scales).  However,  even  these  differences  at  the  extremes 
are  small  in  magnitude  (.2  in  the  root  percent  metric,  .04  in  the  percent  metric)  and  have  little  impact  on 
estimates  of  reported  statistics  such  as  percentages  of  students  below  the  achievement  levels. 


9.7  PRODUCING  A MATHEMATICS  COMPOSITE  SCALE 

For  the  national  assessment,  composite  scales  were  created  for  both  fourth  and  eighth  grade  as 
overall  measures  of  mathematics  scale  scores  for  students  at  that  grade.  The  composite  was  a weighted 
average  of  plausible  values  on  the  five  content  strand  scales  {number  sense , properties , and  operations ; 
measurement;  geometry  and  spatial  sense ; data  analysis,  probability,  and  statistics ; and  algebra  and 
functions).  The  weights  for  the  national  content  strand  scales  were  proportional  to  the  relative  importance 
assigned  to  each  content  strand  for  each  grade  in  the  assessment  specifications  developed  by  the 
Mathematics  Objectives  Panel.  Consequently,  the  weights  for  each  of  the  content  strands  are  similar  to 
the  actual  proportion  of  items  from  that  content  strand  at  each  grade. 

State  Assessment  composite  scales  were  developed  using  weights  identical  to  those  used  to 
produce  the  composites  for  the  1996  national  mathematics  assessment.  The  weights  are  given  in  Table 
9-28.  In  developing  the  State  Assessment  composite  for  each  grade,  the  weights  were  applied  to  the 
plausible  values  for  each  content  strand  scale  as  expressed  in  terms  of  the  final  State  Assessment  scales 
for  each  grade  (i.e.,  after  transformation  from  the  provisional  BILOG/PARSCALE  scales.) 

Figure  9- 1 3 provides  rootograms  comparing  the  estimated  proficiency  distributions  based  on  the 
SA  and  NL  samples  for  the  grade  4 and  grade  8 composites.  Consistent  with  the  results  presented 
separately  by  scale,  there  is  some  evidence  that  the  estimates  produced  using  the  State  Assessment  data 
are  slightly  heavier  in  the  extreme  lower  tails  than  the  corresponding  estimate  based  on  the  NL  data. 
However,  again  these  differences  in  root  relative  percents  are  small  in  magnitude. 


Figure  9-8 

Rootogram  Comparing  Scale  Score  Distributions  for  the  State  Assessment  Aggregate  Sample 
and  the  National  Linking  Sample  for  the  Number  Sense,  Properties,  and  Operati  is  Scale 
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Figure  9-9 

Rootogram  Comparing  Scale  Score  Distributions  for  the  State  Assessment  Aggregate  Sample 
and  the  National  Linking  Sample  for  the  Measurement  Scale 
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Figure  9-10 

Rootogram  Comparing  Scale  Score  Distributions  for  the  State  Assessment  Aggregate  Sample 
and  the  National  Linking  Sample  for  the  Geometry  and  Spatial  Sense  Scale 
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Figure  9-12 

Rootogram  Comparing  Scale  Score  Distributions  for  the  State  Assessment  Aggregate  Sample 
and  the  National  Linking  Sample  for  the  Algebra  and  Functions  Scale 
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Table  9-28 

Weights  Used  for  Each  Scale  to  Form  Grade  4 and  Grade  8 Composites 


Scale 

Grade  4 

Grade  8 

Number  Sense,  Properties,  and  Operations 

.40 

.25 

Measurement 

.20 

.15 

Geometry  and  Spatial  Sense 

.15 

.20 

Data  Analysis,  Statistics,  and  Probability 

.10 

.15 

Algebra  and  Functions 

.15 

.25 

Figure  9-13 

Rootogram  Comparing  Scale  Score  Distributions  for  the  State  Assessment  Aggregate  Sample 
and  the  National  Linking  Sample  for  the  Composite  Scale 
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9.8  THE  WEIGHT  FILES 


Westat  produced  the  final  student  and  school  weights  and  the  corresponding  replicate  weights  for 
the  1996  State  Assessment.  Information  for  the  creation  of  the  weight  files  was  supplied  by  NCS  under 
the  direction  of  ETS.  Because  the  State  Assessment  sample  was  split  into  two  subsamples,  one  using  the 
1992  inclusion  rules  (SI)  and  one  using  the  1996  inclusion  rules  (S2)  the  weighting  process  was  more 
complex  than  in  previous  assessments.  Westat  provided  a total  of  four  files:  a student  and  school  file  for 
each  grade  assessed  in  the  1996  State  Assessment. 

The  student  weight  files  contained  one  record  for  every  student  who  was  not  classified  as  a SD  or 
LEP;  the  weight  files  contained  two  records  for  every  student  who  was  classified  as  SD  or  LEP.  Each 
record  had  a full  set  of  weights,  including  replicate  weights.  The  first  set  of  weights  for  the  SD  and  LEP 
students  is  to  be  used  when  estimating  results  for  either  SI  or  S2  alone.  The  second  set  of  weights 
provided  for  those  students  is  to  be  used  when  estimating  results  for  students  from  both  S 1 and  S2 
together.  (See  Chapters  3 and  7 for  more  information  about  the  sampling  and  weighting  procedures  for 
the  S 1 and  S2  samples.) 

From  the  student  weight  files,  ETS  constructed  three  sets  of  student  weights,  called  modular 
weights,  reporting  weights,  and  all-inclusive  weights.  The  modular  weights  were  used  when  examining 
S 1 and  S2  separately,  or  for  comparing  S 1 to  S2.  The  reporting  weights,  used  for  most  reports,  were  used 
when  reporting  results  for  the  students  who  were  not  classified  as  being  SD  or  LEP  in  both  SI  and  S2 
and  the  students  classified  as  SD  or  LEP  from  SI  only.  The  reporting  sample  was  formed  so  that  valid 
comparisons  with  previous  mathematics  State  Assessments  could  be  made.  (For  science,  only  students 
classified  as  SD  or  LEP  from  S2  were  included  in  the  reporting  sample.)  The  SD/LEP  students  were 
divided  into  two  types,  those  who  were  assessed  and  those  who  could  not  be  assessed  (called  excluded 
students).  The  all-inclusive  weights  were  used  for  estimating  results  for  both  SI  and  S2  together. 

The  reporting  weights  were  formed  from  the  student  weight  files  by  taking  the  records  for 
students  not  classified  as  SD  or  LEP,  the  first  record  for  students  in  SI  classified  as  SD  or  LEP,  and  a 
record  containing  a missing  value  code  for  the  students  in  S2  classified  as  SD  or  LEP.  In  this  way,  the 
new  inclusion  rules  used  with  the  students  classified  as  SD  or  LEP  in  S2  did  not  effect  1996  State 
Assessment  of  mathematics  results.  For  the  modular  weights,  all  students  not  classified  as  SD  or  LEP  had 
their  final  and  replicate  weights  doubled,  while  the  first  record  for  each  SD/LEP  student  was  selected 
directly  from  the  student  weight  files.  It  is  important  to  note  that  the  samples  should  be  separated  into  the 
SI  and  S2  subsamples  when  using  weights  generated  in  this  way.  To  analyze  data  from  SI  and  S2 
together,  the  all-inclusive  weights  should  be  used.  They  were  created  from  the  student  weight  files  by 
taking  the  records  for  the  students  not  classified  as  SD  or  LEP,  and  the  second  records  for  all  students 
classified  as  SD  or  LEP. 

Each  set  of  weights  (modular,  reporting,  house,  senate,  and  all-inclusive  weights)  has  replicate 
weights  associated  with  it.  Replicate  weights  are  used  to  estimate  jackknife  standard  errors  for  each 
statistic  estimated  for  the  State  Assessments. 

For  the  reporting  sample,  two  other  weights  were  created.  These  are  called  “house  weights”  and 
“senate  weights.”  As  the  respective  branches  of  Congress  do,  these  weights  represent  jurisdictions  in  two 
different  ways.  The  house  weights  weight  the  student  records  within  a jurisdiction  so  that  the  sum  of  the 
weights  for  each  jurisdiction  is  proportional  to  the  fraction  of  the  national  in-grade  enrollment  in  that 
jurisdiction.  The  senate  weights  weight  the  student  records  within  a jurisdiction  so  that  the  sums  of  the 
weights  for  each  jurisdiction  are  approximately  equal  to  each  other.  In  other  words,  a jurisdiction,  like 


California,  with  many  eighth-grade  students  and  a jurisdiction,  like  Rhode  Island,  with  many  fewer 
eighth-grade  students  would  have  equal  weight  when  all  of  the  State  Assessment  data  are  combined.  Both 
of  these  sets  of  weights  are  constructed  only  for  the  reporting  sample.  The  reporting  sample  and  either 
the  house  or  senate  weights  are  used  during  scaling,  conditioning  and  all  major  reporting. 

The  house  weight  is  the  student’s  reporting  weight  times  a factor,  which  is  the  number  of  public 
school  students  sampled  pver  the  sum  of  the  reporting  weights  of  the  public  school  students  in  all  the 
jurisdictions.  The  senate  weight  is  calculated  for  each  jurisdiction  separately.  Within  each  jurisdiction  a 
factor,  which  is  2,000  divided  by  the  sum  of  the  reporting  weights  of  the  jurisdiction’s  public  school 
students,  is  computed.  (For  the  1996  State  Assessment,  2,000  rather  than  the  number  of  public  or 
nonpublic  school  students  within  the  jurisdiction  that  was  used  in  previous  State  Assessments,  was  used 
because  of  the  varying  sample  sizes  for  each  jurisdiction.)  The  reporting  weights  for  students  in  both 
public  and  nonpublic  schools  are  multiplied  by  this  factor  to  create  the  senate  weights.  For  Guam  and 
DoDEA  jurisdictions,  all  schools  were  considered  public  in  the  calculation  of  these  factors. 

In  addition  to  student  weights,  school  weights  are  available  for  use  in  school  level  analyses. 
These  weights  are  modular  weights  for  use  when  examining  S 1 and  S2  separately  or  for  comparing  S 1 to 
S2.  No  other  school  weights  are  available.  School  level  statistics  should  be  calculated  on  the  basis  of  SI 
or  S2  subsamples,  as  opposed  to  reporting  samples.  If  school  level  statistics  are  calculated  for  the 
reporting  sample,  biases  might  occur. 
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Chapter  10 


CONVENTIONS  USED  IN  REPORTING  THE  RESULTS  OF  THE 
1996  STATE  ASSESSMENT  PROGRAM  IN  MATHEMATICS1 

Spencer  S.  Swinton,  David  S.  Freund,  and  Clyde  M.  Reese 
Educational  Testing  Service 


10.1  OVERVIEW 

Results  for  the  1996  State  Assessment  in  mathematics  were  disseminated  in  several 
different  reports:  a Mathematics  State  Report  for  each  jurisdiction,  a brief  report  entitled  the 
NAEP  1996  Mathematics  Report  Card  for  the  Nation  and  the  States , the  Cross-State  Data 
Compendium  for  the  NAEP  1996  Mathematics  Assessment,  and,  distributed  only  in  electronic 
form,  a six-section  almanac  of  data  for  each  jurisdiction. 

The  Mathematics  State  Report  is  a computer-generated  report  that  provides,  for  each 
jurisdiction,  mathematics  results  for  its  fourth-grade  and  eighth-grade  students.  Although 
national  and  regional  results2  are  included  for  comparison  purposes,  the  major  focus  of  each  of 
these  computer-generated  reports  is  on  the  results  for  a particular  jurisdiction.  Data  about  school 
and  student  participation  rates  are  reported  for  each  jurisdiction  to  provide  information  about  the 
aeneralizability  of  the  results.  School  participation  rates  are  reported  both  in  terms  of  the  initially 
selected  samples  of  schools  and  in  terms  of  the  finally  achieved  samples,  including  replacement 
schools.  Several  different  student  participation  rates  are  reported,  including  the  overall  rate,  the 
total  percentage  of  students  excluded  from  the  assessment,  and  the  exclusion  rates  for  students 
who  are  identified  as  being  of  limited  English  proficiency  (LEP)  and  for  students  with 
disabilities  (SD).  In  addition  to  1996  results,  the  state  reports  contain  comparisons  of  1992 
fourth-grade  results  to  the  1996  fourth-grade  results  for  the  jurisdictions  that  participated  in  both 
assessments,  and  of  the  1990  and/or  1992  eighth-grade  results  to  the  1996  eighth-grade  results  as 
applicable.  Trend  results  are  also  provided  for  the  nation  and  for  the  relevant  region  associated 
with  each  participating  jurisdiction,  for  the  same  years  as  reported  tor  that  jurisdiction. 

The  State  Report  text  and  tables  were  produced  by  a computerized  report  generation 
system  developed  by  ETS  report  writers,  statisticians,  data  analysts,  graphic  designers,  and 
editors.  Detailed  technical  documentation  about  the  NAEP  computer-generated  reporting  system 
can  be  found  in  the  technical  documentation  of  The  NAEP  Compu'er-Generated  Reporting 
System  for  the  1994  Trial  State  Assessment  (Jerry,  1995).  Additional  information  is  provided  in 
Sections  10.5.7  and  10.5.8.  The  reports  contain  state-level  estimates  of  scale  score  means, 
proportions  of  students  at  or  above  achievement  levels  defined  by  the  National  Assessment 

1 Spencer  S.  Swinton  played  a role  in  making  decisions  about  hypothesis  testing  methods  and  procedures  and  worked 
with  David  S.  Freund  who  implemented  many  of  the  methods  and  procedures  in  computer  programs.  Spencer  and 
David  worked  with  Clyde  M.  Reese  and  others  to  make  decisions  about  the  rules  used  in  generating  stale  reports.  John 
Ma//eo  contributed  significantly  to  previous  versions  of  this  chapter. 

*Tlie  national  and  regional  results  included  in  the  state  reports  and  in  the  portions  of  the  Cross-Suite  Punt 
Compendium  for  the  SAPP  /W>  Mathematics  Assessment  are  based  on  data  from  the  1990.  1992.  and  1996  national 
mathematics  assessments  ami  include  lonrth  and  eighth  glade  students  entolled  in  public  and  nonpublic  schools 


Governing  Board  (NAGB)  and  selected  percentiles  for  the  state  as  a whole  and  for  subgroups 
defined  by  six  key  reporting  variables  (referred  to  here  as  primary  reporting  variables) — gender, 
race/ethnicity,  level  of  parents’  education.  Title  1 participation,  eligibility  for  the  free  or  reduced- 
cost  school  lunch  component  of  the  National  School  Lunch  Program,  and  type  of  location.  For 
jurisdictions  that  secured  a sufficient  level  of  participation  (see  Appendix  B),  means, 
achievement  levels,  and  percentile  results  were  also  reported  for  students  in  nonpublic  schools 
(Catholic  schools,  other  religious  schools,  and  other  private  schools),  and  for  the  total  in-school 
population  (public-school  students,  nonpublic-school  students,  students  from  the  Department  of 
Defense  Domestic  Dependent  Elementary  and  Secondary  Schools  (DDESS)  and  Department  of 
Defense  Dependents  Schools  (DoDDS),  and  students  attending  Bureau  of  Indian  Affairs  (BIA) 
schools).  In  addition,  for  public-school  students,  scale  score  means  were  reported  for  a variety  of 
other  subpopulations  defined  by  responses  to  items  from  the  student,  teacher,  and  school 
questionnaires  and  by  school  and  location  demographic  variables  provided  by  Westat.3 

The  second  report,  the  NAEP  1996  Mathematics  Report  Card  for  the  Nation  and  the 
States  highlights  key  assessment  results  for  the  nation  and  summarizes  results  across  the 
jurisdictions  participating  in  the  assessment.  This  report  contains  composite  scale  score  results 
(scale  score  means,  proportions  at  or  above  achievement  levels,  etc.)  for  the  nation,  for  each  of 
the  four  regions  of  the  country,  and  each  jurisdiction  participating  in  the  State  Assessment,  both 
overall  and  by  the  primary  reporting  variables.  In  addition,  overall  results  are  reported  for  each  of 
the  content  strand  scales.  Additional  reports,  to  be  released  after  the  Report  Card,  will  include 
trend  comparisons  to  1990  and  1992  for  grade  4 for  those  jurisdictions  that  participated  in  both 
the  1992  and  1996  State  Assessments  and  for  grade  8 for  those  jurisdictions  that  participated  in 
the  1990,  1992,  and  1996  State  Assessments. 

The  third  type  of  report  is  entitled  the  Cross-State  Data  Compendium  for  the  NAEP  1996 
Mathematics  Assessment.  Like  the  Report  Card,  the  Compendium  reports  results  for  the  nation 
and  for  all  of  the  jurisdictions  participating  in  the  State  Assessment.  The  Compendium  contains 
most  of  the  tables  included  in  the  Report  Card  and  State  Report  plus  additional  tables  that 
provide  composite  scale  results  for  a large  number  of  secondary  reporting  variables. 

The  fourth  type  of  summary  report  is  a six-section  electronically-delivered  almanac  that 
contains  a detailed  breakdown  of  the  mathematics  scale  score  data  according  to  the  responses  to 
the  student,  teacher,  and  school  questionnaires  for  the  public-school,  nonpublic-school,  and 
combined  populations  as  a whole  and  for  important  subgroups  of  the  public-school  population. 
There  are  six  sections  to  each  almanac: 

The  Distribution  Data  Section  provides  information  about  the  percentages  of 
students  at  or  above  the  three  composite-scale  achievement  levels  (and  below 
basic).  For  the  composite  scale  and  each  mathematics  scale,  this  almanac  also 
provides  selected  percentiles  for  the  public-school,  nonpublic-school,  and  total 
populations  and  for  the  major  demographic  subgroups  of  the  public-school 
population. 


Some  of  these  variables  were  used  by  Westat,  Inc.,  in  developing  the  sampling  frame  for  the  assessment  and  in 
drawing  the  sample  of  participating  schools.- 
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The  Student  Questionnaire  Section  provides  a breakdown  of  the  composite  scale 
score  data  according  to  the  students'  responses  to  questions  in  the  three  student 
questionnaires  included  in  the  assessment  booklets. 

The  Teacher  Questionnaire  Section  provides  a breakdown  of  the  composite  scale 
score  data  according  to  the  teachers*  responses  to  questions  in  the  mathematics 
teacher  questionnaire. 


The  School  Questionnaire  Section  provides  a breakdown  of  the  composite  scale 
score  data  according  to  the  principals'  (or  other  administrators')  responses  to 
questions  in  the  school  characteristics  and  policies  questionnaire. 

The  Scale  Section  provides  a breakdown  of  the  mathematics  score  results  for  the 
five  mathematics  content  strand  scales  (number  sense , properties , and 
operations : measurement:  geometry  and  spatial  sense ; data  analysis . statistics, 
and  probability ; and  algebra  and  functions)  according  to  selected  items  from  the 
questionnaires. 


The  Mathematics  Item  Section  provides  the  response  data  for  each  mathematics 

item  in  the  assessment. 

The  production  of  the  state  reports,  the  Report  Card , the  Cross-State  Data  Compendium , 
and  the  almanacs  required  many  decisions  about  a variety  ot  data  analysis  and  statistical  issues. 

For  example,  given  the  sample  sizes  obtained  for  each  jurisdiction,  certain  categories  of  the 
reporting  variables  contained  limited  numbers  of  examinees.  A decision  was  needed  as  to  what 
constituted  a sufficient  sample  size  to  permit  the  reliable  reporting  of  subgroup  results,  and 
which,  if  anv,  estimates  were  sufficiently  unreliable  to  need  to  be  identified  (or  flagged)  as  a 
caution  to  readers.  As  a second  example,  the  state  report  contained  computer-generated  text  that 
described  the  results  for  a particular  jurisdiction  and  compared  total  and  subgroup  performance 
within  the  jurisdiction  to  that  of  the  nation.  A number  ot  inferential  rules,  based  on  logical  and 
statistical  considerations,  had  to  be  developed  to  ensure  that  the  computer-generated  reports  were 
coherent  from  a substantive  standpoint  and  were  based  on  statistical  principles  of  significance 
testing.  As  a third  example,  the  Report  Card  contained  tables  that  statistically  compared  eighth- 
g rade^per forman ce  between  1996,  1992,  and  1990  for  each  of  the  participating  jurisdictions. 
Practical  comparison  procedures  were  required  to  control  for  Type  I errors  without  paying  too 
large  a penalty  with  respect  to  the  statistical  power  for  detecting  real  and  substantive  differences. 
For  sets  of  comparisons  w ith  ver>  large  family  sizes,  such  as  the  state  to  all  other  states,  a new' 
multiple  comparison  criterion.  False  Discovers  Rate  or  FDR  (Benjamini  <fc  Hochberg,  1994).  was 
implemented,  which  controls  the  rate  of  false  rejections  (e.g.,  five  false  rejections  per  100 
rejections),  rather  than  controlling  the  probability  of  even  one  such  error  (Familywise  Error  Rate, 
or  FWH).  as  does  the  Bonlerrom  procedure. 

The  purpose  of  this  chapter  is  to  document  the  major  conventions  and  statistical 
procedures  used  in  generating  the  state  reports,  the  Report  Card , the  Cross-State  Data 
Compendium , and  the  almanacs.  The  principal  focus  of  this  chapter  is  on  conventions  used  in  the 
production  of  the  computer-generated  state  reports.  However,  Sections  10.2  to  10.4  contain 
material  applicable  to  all  reports.  Additional  details  about  procedures  relevant  to  the  Report  Card 
and  Cross-State  Data  Compendium  can  be  found  in  the  text  and  technical  appendices  of  those 
ic ports.  Specific  guidelines  fot  the  publication  and  notation  of  NAHP  results  can  be  found  in 
Appendix  B. 
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10.2  MINIMUM  SCHOOL  AND  STUDENT  SAMPLE  SIZES  FOR 
REPORTING  SUBGROUP  RESULTS 

In  all  of  the  reports,  estimates  of  quantities  such  as  composite  and  content  strand  scale 
score  means,  percentages  of  students  at  or  above  the  achievement  levels,  and  percentages  of 
students  indicating  particular  levels  of  background  variables  (as  measured  in  the  student,  teacher, 
and  school  questionnaires)  are  reported  for  the  population  of  students  in  each  jurisdiction  and 
grade,  as  well  as  for  certain  key  subgroups  of  interest.  The  key  subgroups  were  defined  by  six 
primary  NAEP  reporting  variables.  Where  possible,  NAEP  reports  results  for  gender,  for  five 
racial/ethnic  subgroups  (White,  Black,  Hispanic,  Asian  American/Pacific  Islander,  and  American 
Indian/Alaskan  Native),  three  types  of  locations  (central  cities,  urban  fringes/large  towns, 
rural/small  town  areas),  four  levels  of  parents’  education  (did  not  finish  high  school,  high  school 
graduate,  some  college,  college  graduate),  Title  1 participation,  and  eligibility  for  the  free  or 
reduced-cost  school  lunch  component  of  the  National  School  Lunch  Program.  However,  in  some 
jurisdictions,  and  for  some  regions  of  the  country,  school  and/or  student  sample  sizes  were  quite 
small  for  one  or  more  of  the  categories  of  these  variables.  One  would  expect  results  for  these 
subgroups  to  be  imprecisely  estimated. 

It  is  common  practice  in  reports  generated  by  statistical  agencies  to  suppress  those 
estimates  for  which  the  sampling  error  is  so  large  that  it  is  determined  that  no  effective  use  can 
be  made  of  the  estimate,  or  that  the  potential  for  misinterpretation  outweighs  potential  benefits  of 
presenting  results.  A second,  and  equally  important,  consideration  is  whether  the  standard  error 
estimate  that  accompanies  a statistic  is  sufficiently  accurate  to  inform  potential  readers  about  the 
reliability  of  the  statistic.  The  precision  of  a sample  estimate  (be  it  sample  mean  or  standard  error 
estimate)  for  a population  subgroup  from  a two-stage  sample  design  (such  as  was  used  to  select 
the  samples  for  the  State  Assessment)  is  a function  of  the  sample  size  of  the  subgroup  and  of  the 
distribution  of  that  sample  across  first-stage  sampling  units  (i.e.,  schools  in  the  case  of  the  State 
Assessment).  Hence,  both  of  these  factors  were  used  in  establishing  minimum  sample  sizes  for 
reporting. 

For  results  to  be  reported  for  any  subgroup,  a minimum  student  sample  size  of  62  was 
required.  This  number  was  obtained  by  determining  the  sample  size  necessary  to  detect  an  effect 
size  of  .5  with  a probability  of  .8  or  greater4.  The  effect  size  of  .5  pertains  to  the  “true”  difference 
in  mean  scale  scores  between  the  subgroup  in  question  and  the  total  fourth-  or  eighth-grade 
public-school  population  in  the  jurisdiction,  divided  by  the  standard  deviation  of  scale  scores  in 
the  total  population.  The  same  convention  was  used  in  reporting  the  1990,  1992,  and  1996  State 
Assessment  results.  Furthermore,  it  was  required  that  the  students  within  a subgroup  be 
adequately  distributed  across  schools  to  allow  for  reasonably  accurate  estimation  of  standard 
errors.  In  consultation  with  Westat,  a decision  was  reached  to  publish  only  those  statistics  that 
had  standard  errors  estimates  based  on  five  or  more  degrees  of  freedom.  Slightly  different 
variance  estimation  procedures  were  used  to  obtain  estimated  standard  errors  for  public-  and 
nonpublic-school  statistics  (see  Chapter  7).  These  different  procedures  implied  different 


4 A design  effect  of  2 was  assumed  for  this  purpose,  implying  a sample  design-based  variance  twice  that  of  simple 
random  sampling.  This  is  consistent  with  previous  NAEP  experience  (Johnson  & Rust,  1992).  In  carrying  out  the 
statistical  power  calculations  when  comparing  a subgroup  to  the  total  group,  it  was  assumed  that  the  total  population 
sample  si/c  is  large  enough  to  make  a negligible  contribution  to  standard  errors. 
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minimum  school  sample  sizes  for  public-  and  nonpublic-school  results  in  order  to  meet  the  five 
degrees  of  freedom  minimum.  For  public-school  statistics,  subgroup  data  were  required  to  come 
from  a minimum  of  five  stratification  categories  formed  for  variance  estimation.  For  nonpublic- 
school  statistics,  a six-school  minimum  was  required. 

It  should  be  noted  that  the  full  set  of  reports  includes  large  numbers  of  tables  that  provide 
estimates  of  the  proportion  of  the  students  responding  to  each  category  of  a secondary  reporting  , 
variable,  as  well  as  the  mean  scale  scores  of  the  students  within  each  category.  In  several 
instances,  the  number  of  students  in  a particular  category  of  these  background  variables  was  also 
less  than  62  or  was  clustered  within  a small  number  of  schools.  The  same  minimum  student  and 
school  sample  size  restrictions  were  applied  in  the  case  of  proportions  as  were  used  for  scale 
score  means. 


10.3  ESTIMATES  OF  STANDARD  ERRORS  WITH  LARGE  MEAN 
SQUARED  ERRORS 


As  noted  above,  standard  errors  of  mean  scale  scores,  proportions,  and  percentiles  play 
an  important  role  in  interpreting  subgroup  results  and  in  comparing  the  performances  of  two  or 
more  subgroups.  The  jackknife  standard  errors  reported  by  NAEP  are  statistics  whose  quality 
depends  on  certain  features  of  the  sample  from  which  the  estimate  is  obtained.  As  discussed  in 
the  previous  section,  in  certain  cases,  typically  when  the  number  of  students  upon  which  the 
standard  error  is  based  is  small  or  when  this  group  of  students  come  from  a small  number  of 
participating  schools,  the  mean  squared  error5  associated  with  the  estimated  standard  errors  may 
be  quite  large.  Minimum  school  and  student  sample  sizes  were  implemented  which  suppressed 
statistics  in  most  instances  where  such  problems  existed.  However,  the  possibility  remained  that 
some  statistics  based  on  sample  sizes  that  exceed  the  minimum  requirements  might  still  be 
associated  with  standard  errors  that  were  not  well  estimated.  Therefore,  in  the  reports,  estimated 
standard  errors  for  published  statistics  that  are  subject  to  large  mean  squared  errors  are  followed 
by  the  symbol  “!”. 


The  magnitude  of  the  mean  squared  error  associated  with  an  estimated  standard  error  for 
the  mean  or  proportion  of  a group  depends  on  the  coefficient  of  variation  ( CV)  of  the  estimated 
size  of  the  population  group,  denoted  as  N (Cochran,  1977,  Section  6.3).  The  coefficient  of 
variation  is  estimated  by: 


CV(N)  = 


SE(N) 

N 


where  N is  a point  estimate  of  N and  SE(N)  is  the  jackknife  standard  error  (described  in 

A 

Chapter  10  of  The  NAEP  1994  Technical  Report,  Alien,  Kline,  & Zelenak,  1996)  of  N. 

Experience  with  previous  NAEP  assessments  suggests  that  when  this  coefficient  exceeds 
.2,  the  mean  squared  error  of  the  estimated  standard  errors  of  means  and  proportions  based  on 
samples  of  this  size  may  be  quite  large.  (Further  discussion  of  this  issue  can  be  found  in  Johnson 


^Thc  mean  squared  error  of  the  estimated  standard  error  is  defined  as  T IS  ■ a [.  where  5 is  the  estimated  standard 
error.  O' is  the  “true"  standard  error,  and  '6  is  the  expectation,  or  expected  value  operator. 
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& Rust,  1992.)  Therefore,  the  standard  errors  of  means  and  proportions  for  all  subgroups  for 
which  the  coefficient  of  variation  of  the  population  size  exceeds  .2  are  followed  by  “!”  in  the 
tables  of  all  reports.  These  standard  errors,  and  any  confidence  intervals  or  significance  tests 
involving  these  standard  errors,  should  be  interpreted  with  caution.  In  the  Report  Card , the 
Cross-State  Data  Compendium , and  the  almanacs,  statistical  tests  involving  one  or  more 
quantities  that  have  standard  errors  so  Hugged  should  be  interpreted  with  caution. 


10.4  TREATMENT  OF  MISSING  DATA  FROM  THE  STUDENT,  TEACHER, 
AND  SCHOOL  QUESTIONNAIRES 

Responses  to  the  student,  teacher,  and  school  questionnaires  played  a prominent  role  in 
all  reports.  Although  the  return  rate  on  all  three  types  of  questionnaire  was  high,6  there  were 
missing  data  for  each  type  of  questionnaire. 

The  reported  estimated  percentages  of  students  in  the  various  categories  of  background 
variables,  and  the  estimates  of  the  mean  scale  score  of  such  groups,  were  based  on  only  those 
students  for  whom  data  on  the  background  variable  were  available.  In  the  terminology  of  Little 
and  Rubin  ( 1987),  the  analyses  pertaining  to  a particular  background  variable  presented  in  the 
state  reports  and  the  Cross -State  Data  Compendium  assume  the  data  are  missing  completely  at 
random  (i.e.,  the  mechanism  generating  the  missing  data  is  independent  of  both  the  response  to 
the  particular  background  items  and  the  scale  score). 

The  estimates  of  proportions  and  proficiencies  based  on  “missing-completely-at-random” 
assumptions  are  subject  to  potential  nonresponse  bias  if,  as  may  be  the  case,  the  assumptions  are 
not  correct.  The  amount  of  missing  data  was  small  (usually,  less  than  2%)  for  most  of  the 
variables  obtained  from  the  student  and  school  questionnaires.  For  analyses  based  on  these 
variables,  reported  results  are  subject  to  little,  if  any,  nonresponse  bias.  However,  for  particular 
background  items  from  the  student  and  school  questionnaires,  the  level  of  nonresponse  in  certain 
jurisdictions  was  somewhat  higher.  As  a result,  the  potential  for  nonresponse  bias  in  the  results 
of  analyses  based  on  this  latter  set  of  background  items  is  also  somewhat  greater.  Background 
items  for  which  more  than  10  percent  of  the  returned  questionnaires  were  missing  are  identified 
in  background  almanacs  produced  for  each  jurisdiction.  Again,  results  for  analyses  involving 
these  items  should  be  interpreted  with  caution. 

In  order  to  analyze  the  relationships  between  teachers'  questionnaire  responses  and  their 
students'  achievement,  each  teacher's  questionnaire  had  to  be  matched  to  the  students  who  were 
taught  mathematics  by  that  teacher.  Table  10-1  provides  the  percentages  of  fourth-grade  students 
that  were  matched  to  teacher  questionnaires  for  each  of  the  47  jurisdictions  that  participated  in 
the  State  Assessment.  Table  10-2  contains  similar  information  for  the  eighth-grade  sample.  In 
both  tables,  the  first  column  presents  match  rates  for  public-school  students  and  the  second  for 
nonpublic-school  students.  Note  that  these  match  rates  do  not  reflect  the  additional  missing  data 
due  to  item-level  nonresponse.  The  amount  of  additional  item-level  nonresponse  in  the  returned 
teacher  questionnaires  can  be  found  in  the  almanacs  produced  for  each  jurisdiction. 


' Infot m.iiinn  about  Miru'\  participant*!  rales  ihoib  n,  hnol  .mil  student  i.  as  \sell  as  proportions  ol  students  excluded  by 
each  juusdictinn  Irnm  the  assessment . arc  t.-ivcn  in  \ppciuii\  B Adjustments  intended  to  account  for  school  and 
student  nonrc.poii «’  aic  dcst.  ubed  in  t haptn 
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Table  10-1 


Weighted  Percentage  of  Fourth-Grade  Students  Matched  to  Teacher  Questionnaires 


Jurisdiction 

Public 

Nonpublic 

Alabama 

97 

100 

Alaska 

86 

— 

Arizona 

89 

92 

Arkansas 

97 

100 

California 

93 

98 

Colorado 

97 

94 

Connecticut 

98 

100 

Delaware 

95 

90 

District  of  Columbia 

93 

83 

DoDEA/DDESS 

95 

-- 

DoDEA/DoDDS 

94 

- 

Florida 

96 

96 

Georgia 

98 

97 

Guam 

89 

84 

Hawaii 

95 

- 

Indiana 

97 

91 

Iowa 

95 

88 

Kentucky 

94 

90 

Louisiana 

99 

100 

Maine 

95 

100 

Maryland 

97 

100 

Massachusetts 

97 

90 

Michigan 

97 

97 

Minnesota 

96 

92 

Mississippi 

96 

89 

Missouri 

91 

99 

Montana 

93 

100 

Nebraska 

97 

86 

Nevada 

97 

100 

New  Jersey 

98 

89 

New  Mexico 

92 

68 

New  York 

97 

91 

North  Carolina 

96 

-- 

North  Dakota 

95 

84 

Oregon 

95 

75 

Pennsylvania 

97 

94 

Rhode  Island 

99 

— 

South  Carolina 

97 

— 

Tennessee 

97 

— 

Texas 

92 

100 

Utah 

94 

100 

Vermont 

98 

98 

Virginia 

97 

— 

Washington 

97 

West  Virginia 

95 

— 

Wisconsin 

94 

100 

Wyoming 

95 

100 

O 7 O 

6 1 t- 
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Table  10-2 

Weighted  Percentage  of  Eighth-Grade  Students  Matched  to  Teacher  Questionnaires 


Jurisdiction 

Public 

Nonpublic 

Alabama 

95 

98 

Alaska 

74 

Arizona 

90 

— 

Arkansas 

95 

100 

California 

90 

85 

Colorado 

89 

— 

Connecticut 

90 

94 

Delaware 

91 

100 

District  of  Columbia 

74 

92 

DoDEA/DDESS 

100 

- 

DoDEA/DoDDS 

95 

- 

Florida 

91 

- 

Georgia 

94 

100 

Guam 

56 

82 

Hawaii 

88 

- 

Indiana 

93 

- 

Iowa 

95 

99 

Kentucky 

86 

91 

Louisiana 

90 

94 

Maine 

91 

— 

Maryland 

94 

95 

Massachusetts 

92 

92 

Michigan 

92 

99 

Minnesota 

95 

94 

Mississippi 

92 

- 

Missouri 

92 

94 

Montana 

88 

100 

Nebraska 

95 

88 

Nevada 

88 

100 

New  Hampshire 

-- 

92 

New  Jersey 

96 

90 

New  Mexico 

.97 

100 

New  York 

89 

94 

North  Carolina 

86 

- 

North  Dakota 

96 

90 

Oregon 

92 

100 

Pennsylvania 

87 

- 

Rhode  Island 

94 

99 

South  Carolina 

92 

87 

Tennessee 

94 

- 

Texas 

97 

98 

Utah 

91 

100 

Vermont 

92 

100 

Virginia 

94 

- 

Washington 

88 

79 

West  Virginia 

96 

- 

Wisconsin 

88 

93 

Wyoming 

95 

1(H) 
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10.5  STATISTICAL  RULES  USED  FOR  PRODUCING  THE  STATE  REPORTS 


As  described  earlier,  the  state  reports  contain  jurisdiction-level  estimates  of  fourth-  and 
eighth-grade  mean  proficiencies,  proportions  of  students  at  or  above  selected  scale  points,  and 
percentiles  for  the  jurisdiction  as  a whole  and  for  the  categories  of  a large  number  of  reporting 
variables.  Similar  results  are  provided  for  the  nation  and,  where  sample  sizes  permitted,  for  the 
region  to  which  each  jurisdiction  belongs7.  The  state  reports  were  computer-generated.  The 
tables  and  Figures,  as  well  as  the  text  of  the  report,  were  automatically  tailored  for  each 
jurisdiction  based  on  the  pattern  of  results  obtained.  The  purpose  of  this  section  is  to  describe 
some  of  the  procedures  and  rules  used  to  produce  these  individualiy  tailored  reports.  A detailed 
presentation  is  available  in  the  technical  documentation  of  The  NAEP  Computer-Generated 
Reporting  System  for  the  1994  Trial  State  Assessment  (Jerry,  1995).  Some  changes  were  made 
for  the  1996  State  Assessment,  and  the  current  procedures  and  rules  for  State  Reports  are 
documented  in  this  chapter. 

In  the  1996  state  reports,  the  results  are  presented  principally  through  a sequence  of 
tables  containing  estimated  means,  proportions,  and  percentiles,  along  with  their  standard  errors, 
for  1996  and,  where  appropriate,  for  1992  and  1990.  In  addition  to  the  tables  of  results, 
computer-generated  interpretive  text  is  also  provided.  In  some  cases,  the  computer-generated 
interpretive  text  is  primarily  descriptive  in  nature  and  reports  the  total  group  and  subgroup  scale 
score  means  and  proportions  of  interest.  However,  some  of  the  interpretive  text  focuses  on 
interesting  and  potentially  important  group  differences  in  mathematics  scale  scores  or  on  the 
percentages  of  students  responding  in  particular  ways  to  the  background  questions.  Additional 
interpretive  text  compares  state-level  results  with  those  of  the  nation,  and  discusses  changes  in 
results  from  1992  and  1990  to  1996.  For  example,  one  question  of  interest  to  each  jurisdiction  is 
whether,  on  average,  its  students  performed  higher  than,  lower  than,  or  about  the  same  as 
students  in  the  nation.  Additional  interpretive  text  focuses  on  patterns  of  achievement  across  the 
mathematics  content  strands  or  on  the  pattern  of  response  to  a particular  background  item  in  the 
jurisdiction.  For  example,  do  more  students  report  spending  30  minutes  or  1 5 minutes  on 
homework  each  day? 

Rules  were  developed  to  produce  the  computer-generated  text  for  comparisons  of  results 
for  subgroups  and  for  interpretations  of  patterns  of  results.  These  rules  were  based  on  a variety  of 
considerations,  including  a desire  for  1)  statistical  rigor  in  the  identification  of  important  group 
differences  and  patterns  of  results,  and  2)  solutions  that  were  within  the  limitations  imposed  by 
the  availability  of  computational  resources  and  the  time  frame  for  the  production  of  the  report. 
The  following  sections  describe  some  of  these  procedures  and  rules. 


’Because  IJ.S.  territories  arc  not  classified  into  NAEP  regions,  no  regional  comparisons  arc  provided  for  Guam 
Regional  results  arc  also  not  provided  for  the  DDESS  and  DoDDS  schools. 


10.5.1  Comparing  Means  and  Proportions  for  Different  Groups  of  Students 

Many  of  the  group  comparisons  explicitly  commented  on  in  the  state  reports  involved 
mutually  exclusive  sets  of  students.  One  common  example  of  such  a comparison  is  the  contrast 
between  the  mean  composite  score  in  a particular  jurisdiction  and  the  mean  composite  score  in 
the  nation.  Other  examples  include  comparisons  within  a jurisdiction  of  the  average  scale  score 
for  male  and  female  students;  White  and  Hispanic  students;  students  attending  schools  in  central 
city  and  urban  fringe/large  town  locations;  and  students  who  reported  watching  six  or  more  hours 
of  television  each  night  and  students  who  report  watching  less  than  one  hour  each  night. 

In  the  state  reports,  computer-generated  text  indicated  that  means  or  proportions  from 
two  groups  were  different  only  when  the  difference  in  the  point  estimates  for  the  groups  being 
compared  was  statistically  significant  at  an  approximate  simultaneous  a level  of  .05.  An 
approximate  procedure  was  used  for  determining  statistical  significance  NAEP  staff  judged  to  be 
statistically  defensible,  as  well  as  being  computationally  tractable.  Although  all  pairs  of  levels 
within  a variable  were  tested,  computer-generated  text  was  developed  for  only  a subset  of  the 
possible  comparisons.  For  example,  text  was  included  to  compare  the  majority  ethnic  group  and 
each  minority  group,  but  text  for  all  possible  comparisons  of  groups  was  not  included,  even  if 
some  unreported  comparisons  were  significant.  The  procedure  used  to  make  statistical  tests  is 
described  in  the  following  paragraphs. 

Let  Ai  be  the  statistic  in  question  (i.e.,  a mean  for  group  i)  and  let  SAi  be  the  jackknife 
standard  error  of  the  statistic.  The  computer-generated  text  in  the  state  report  identified  the 
means  or  proportions  for  groups  i and  j as  being  different  if  and  only  if: 


where  Ta  is  the  (1  - a)  percentile  of  the  distribution  with  degrees  of  freedom,  df,  as  estimated 
below,  and  c is  the  number  of  related  comparisons  being  tected.  See  Section  10.5.2  for  a more 
specific  description  of  multiple  comparisons.  In  cases  where  group  comparisons  were  treated  as 
individual  units  (for  example,  comparing  overall  state  results  with  overall  national  results  or 
overall  state  results  in  1996  with  those  of  1992,  the  value  of  c was  taken  as  1,  and  the  test 
statistic  was  approximately  equivalent  to  a standard  two-tailed  t-test  for  the  difference  between 
group  means  or  proportions  from  large  independent  samples  with  the  a level  set  at  .05. 

In  some  cases,  it  was  desired  to  test  the  change  in  the  size  of  the  difference  between  two 
groups  from  one  assessment  time  to  the  next.  For  example,  suppose  White  students  in  1992  had 
mean  scale  score  A , with  jackknife  standard  error  SAl,  and  in  1996,  mean  scale  score  A2,  with 
jackknife  standard  error  Sai  • Further,  suppose  Hispanic  students  in  1992  had  mean  scale  score  A? 
with  jackknife  standard  error  SAy  and  in  1996,  mean  scale  score  A4,  with  jackknife  standard  error 
SA  Then  to  test  whether  the  difference  between  the  two  groups  has  increased  significantly,  has 
decreased  significantly,  or  remains  about  the  same,  the  following  t-test  was  used: 

(A,  — A,)-(A2-A4) 
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The  degrees  of  freedom  for  both  t-tests  of  differences  is  defined  by  a Satterthwaite  (Johnson  & 
Rust,  1992)  approximation  as  follows: 
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where  N is  the  number  of  subgroups  involved,  four  in  this  case,  and  df  is  as  follows: 


<n\ 


til 


where  m is  the  number  of  nonempty  PSU  pairs  for  statistic  k.  / : is  the  jth  replicate  estimate 
(pseudo  value)  for  the  mean  of  subgroup  k , and  / is  the  estimate  of  the  subgroup  mean  using  the 
overall  weight  and  the  first  plausible  value  for  subgroup  k.  Flic  number  of  nonempty  PSU  pairs  is 
so  m is  less  than  or  equal  to  //,  the  total  number  of  PSU  pairs.  The  first  factor  in  the  expression 
dj\  is  an  empirically  based  adjustment  for  the  downward  bias  of  the  degrees  of  freedom  estimate. 
The  degrees  of  freedom  will  be  bounded  as  not  to  be  greater  than  the  number  of  PSU  pairs. 

The  procedures  in  this  section  assume  that  the  data  being  compared  are  from  independent 
samples.  Because  of  the  sampling  design  used  for  the  State  Assessment,  in  which  both  schools 
and  students  within  schools  are  randomly  sampled,  the  data  from  mutually  exclusive  sets  of 
students  within  a jurisdiction  may  not  be  strictly  independent.  Therefore,  the  significance  tests 
employed  are,  in  many  cases,  only  approximate.  As  described  in  the  Section  10.5.4,  another 
procedure,  one  that  does  not  assume  independence,  could  have  been  conducted.  However,  that 
procedure  is  computationally  burdensome  and  resources  precluded  its  application  for  all  the 
comparisons  in  the  state  reports.  It  was  the  judgment  of  NAHP  staff  that  if  the  data  were 
correlated  across  groups,  in  most  cases  the  correlation  was  likely  to  be  positive.  Since,  in  such 
instances,  significance  tests  based  on  assumptions  of  independent  samples  are  conservative 
(because  the  estimated  standard  error  of  the  difference  based  on  independence  assumptions  is 
larger  than  the  more  complicated  estimate  based  on  correlated  croups),  the  approximate 
procedure  was  used  for  most  comparisons. 

The  procedures  described  above  were  used  for  testing  differences  of  both  means  and 
nonextreme  percents.  The  approximation  foi  the  tost  for  percentages  works  best  when  sample 
sizes  are  large,  and  the  percentages  being  tested  have  magnitude  relatively  close  to  50  percent. 
Statements  about  group  differences  should  be  interpreted  with  caution  if  at  least  one  of  the 
groups  being  compared  is  small  in  si/e  and/or  if  "extreme'*  percentages  are  being  compared. 
Percentages,  l\  were  treated  as  "extreme"  if 
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P < Plim  = , where  the  effective  sample  size,  NEFF  = — — — 2 — , and  SE  is  the 

Neff+2  (jc) 

jackknife  standard  error  of  P.  This  “rule  of  thumb”  cutoff  leads  to  flagging  a large  proportion  of 
confidence  intervals  that  would  otherwise  include  values  < 0 or  > 1 . Similarly,  at  the  other  end  of 
the  0 - 100  scale,  a percentage  is  deemed  extreme  if  100  - P < PUm.  In  either  extreme  case,  the 
normal  approximation  to  the  distribution  is  a poor  approximation,  and  the  value  of  P was 
reported,  but  no  standard  error  was  estimated  and  hence  no  tests  were  conducted. 


10.5.2  Multiple  Comparison  Procedures 

Frequently,  groups  (or  families)  of  comparisons  were  made  and  were  presented  as  a 
single  set.  The  appropriate  text,  usually  a set  of  sentences  or  a paragraph,  was  selected  for 
inclusion  in  the  report  based  on  the  results  for  the  entire  set  of  comparisons.  For  example, 
Chapter  2 of  the  state  report  contains  a section  that  compared  average  scale  scores  for  a 
predetermined  group,  generally  the  majority  group  (in  the  case  of  race/ethnicity,  for  example. 
White  students)  to  those  obtained  by  other  minority  groups.  For  families  of  comparisons  like 
these,  a Bonferroni  procedure  (Miller,  1966),  controlling  the  Familywise  Error  (FWE)  Rate,  was 
used  for  determining  the  value  of  Ta,  where  c was  the  number  of  contrasts  in  the  set.  In  this 
example,  c was  taken  to  be  the  number  of  minority  groups  meeting  minimum  sample  size 
requirements,  and  each  statistical  test  was  consequently  carried  out  at  an  a level  of  .05/c. 


However,  in  an  attempt  to  gain  greater  power,  two  separate  definitions  of  family  size 
were  employed  for  comparisons  in  two-way  tables.  For  n levels  of  a control  variable  (e.g., 
ethnicity)  and  m levels  of  a comparison  variable  (e.g.,  number  of  hours  of  homework),  the 
standard  Bonferroni  family  size  ofnxmx  (m-l)/2  was  used.  In  addition,  when  the  m x (m-l)/2 
marginal  tests  yielded  a significant  difference  for  a pair  of  categories  of  the  comparison  variable, 
the  n levels  of  the  control  variable  corresponding  to  that  pair  of  categories  were  tested  with  a 
family  size  of  n.  Significance  was  reported  if  either  definition  of  family  size  met  the  criterion. 
Further,  2x2  interactions  were  tested  for  a mxn  table  with  t-tests  using  a family  size 
n x (n-1)  X m x (m- 1 )/4.  In  these  cases,  a modification  due  to  Hochberg  of  the  standard 
Bonferroni  procedure  was  employed,  in  which  probabilities  associated  with  outcomes  are 
ordered,  and  a is  divided  by  an  integer  which  increases  from  1 to  the  family  size  as  successively 
smaller  probabilities  are  tested.  More  formally,  the  Hochberg  Stagewise  Procedure  (Hochberg, 
1988)  is  defined: 

Let  m be  the  number  of  significance  tests  made  (the  family  size)  and  let 
P,  < P2  < ...  < Pm  be  the  ordered  significance  levels  for  the  m tests.  Let  a be  the  combined 

significance  level.  The  Hochberg  procedure  compares  Pm  with  a,  P,„-i  with  a/2 P,  with 

a/(m-j+l),  stopping  comparisons  with  the  first  j such  that  P,  < a/(m-j+l).  All  tests  associated 

with  Pi P,  are  declared  significant,  all  tests  associated  with  P,+, Pm  are  declared 

nonsignificant. 

To  compare  the  jurisdiction  in  a State  Assessment  report  with  the  nation  and  all  other 
participating  jurisdictions,  as  is  done  in  the  comparisons  of  overall  scale  score  maps  in  the  State 
Assessment  reports,  as  many  as  46  different  comparisons  need  to  be  computed.  In  this  case,  the 
multiple  comparison  procedure  (Benjamini  and  Hochberg,  1994)  was  used  to  judge  significance. 
Unlike  the  Bonferroni  procedure  that  controls  the  FWE,  the  procedure  controls  the  expected 


proportion  of  falsely  rejected  hypotheses  among  all  rejections.  For  example,  at  the  .05  level,  for 
every  100  rejections  of  the  null  hypothesis,  the  procedure  ensures  that  no  more  than  five  will  be 
expected  to  be  false. 


The  Benjamini  and  Hochberg  application  of  the  False  Discovery  Rate  (FDR)  critenon 
can  be  described  as  follows.  Let  m be  the  number  of  significance  tests  made  and  let 
Pi  < P2  < • • • < Pm  be  the  ordered  significance  levels  of  the  m tests,  from  lowest  to  highest 
probability.  Let  a be  the  combined  significance  level  desired,  usually  .05.  The  procedure  will 
compare  Pm  with  a,  P^i  with  a(m-l)//n, . . .,  Pj  with  aj/my  stopping  the  comparisons  with  the  first 
j such  that  Pj  < ajfm.  All  tests  associated  with  Pj, . . .,  Pj  are  declared  significant;  all  tests 
associated  with  Pj+i, . . Pm  are  declared  nonsignificant. 

10,5.3  Comparisons  of  1996, 1992,  and  1990  Results  in  the  Mathematics  Report  Card 
and  the  Cross-State  Data  Compendium 

The  Mathematics  Report  Card  and  the  Cross-State  Data  Compendium  contain  many 
tables  that  compare  fourth-and  eighth-grade  public-school  results  for  1996  with  those  obtained  in 
1990  for  grade  8 and  1992  for  grades  4 and  8.  Comparisons  are  also  made  of  results  for  the 
nation  as  a whole,  for  each  of  the  four  regions  of  the  country,  and  for  each  of  the  40  grade  4 and 
37  grade  8 jurisdictions  assessing  public-school  students  that  participated  in  both  the  1996  and 
1992  State  Assessments.  For  the  three-point  trend,  from  1990  through  1992  to  1996,  seven  fewer 
jurisdictions  had  participated  in  the  full  series,  but  one  additional  state  appeared  in  1990  and 
1996  assessments  only.  The  national  and  regional  results  are  based  on  the  1990,  1992,  and  1996 
national  NAEP  public-school  samples.  The  results  for  the  jurisdictions  are  based  on  the  1996, 
1992,  and  1990  State  Assessment  samples.  Each  jurisdiction’s  overall  results  are  compared,  as 
well  as  the  results  for  both  primary  and  secondary  NAEP  reporting  subgroups.  The  following 
statistics  are  compared: 

• the  proportions  of  examinees  in  the  various  primary  and  secondary  reporting 
subgroups; 

• average  scale  scores,  overall  and  for  the  primary  reporting  subgroups,  on  the 
composite  scale,  and  the  mathematics  content  strand  scales; 

• selected  percentiles  (10th,  25th,  50th,  75th,  90th)  overall,  for  the  NAEP 
composite  scale  and  for  the  mathematics  contents  strand  scales;  and 

• proportions  of  students  at  or  above  the  achievement  levels,  overall  and 
within  the  primary  reporting  subgroups,  on  the  composite  scale. 

A number  of  different  types  of  tables  are  included  in  these  reports.  For  example,  one  type 
of  table  shows  the  average  composite  scale  score  and  the  percentage  of  students  at  or  above  each 
of  the  achievement  levels.  A second  type  of  table  shows  the  percentage  of  students  at  or  above 
achievement  levels  on  the  composite  scales  for  each  of  the  primary  reporting  subgroups.  A third 
type  of  table  shows  average  mathematics  scale  score  and  five  percentile  locations  for  each  of  the 
mathematics  content  strand  scales.  A fourth  type  of  table  shows  average  composite  proficiencies 
for  a particular  set  of  primary  or  secondary  reporting  subgroups. 


Because  of  the  large  volume  of  tables  in  the  Mathematics  Report  Card  and  the  Cross- 
State  Data  Compendium,  most  were  computer-generated.  To  help  readers  focus  on  important 
outcomes,  each  of  the  tables  containing  results  for  1996,  1992,  and  1990  are  annotated  with 
symbols  indicating  which  1996-to-1992  or  1996-to- 1990  jurisdiction  comparisons  represent 
statistically  significant  changes8.  The  annotations  to  these  tables  were  made  automatically  by  the 
computer  programs  that  produced  them  and  were  based  on  tests  of  statistical  significance  and 
FWE  or  FDR  criteria  like  those  described  in  Sections  10, 1 and  10.5.2.  This  section  describes  the 
rules  and  conventions  used  by  the  computer  programs  in  annotating  the  tables.  These  rules  and 
conventions  were  chosen  based  on  feasibility  considerations  and  a desire  to  balance  statistical 
power  with  Type  I error  control  within  these  feasibility  constraints. 

Tw'o  types  of  annotations  were  made.  The  first  type  of  annotation  was  used  for  the 
FWE  criterion  to  indicate  a gain  or  loss  that  was  statistically  significantly  different  from  1990, 
considering  each  jurisdiction  as  a separate  entity  and  controlling  for  the  number  of  tests 
conducted  in  a particular  table  within  that  jurisdiction.  Similarly,  the  annotation  w'as  used  to 
indicate  a gain  or  loss  that  was  statistically  significantly  different  from  1992,  considering  each 
jurisdiction  as  a separate  entity  and  controlling  for  the  number  of  tests  conducted  in  a particular 
table  within  that  jurisdiction.  Since  all  tables  were  sei  up  with  jurisdictions  as  the  row'  variable, 
the  first  type  of  annotation  was  used  on  significance  tests  that  separately  controlled  the  Type  I 
error  rate  within  each  row  of  the  table  using  the  Bonferroni  procedure.  The  second  type  of 
annotation  “**”  was  used  to  indicate  a gain  or  loss  that  was  statistically  significantly  different 
from  1990  after  simultaneously  controlling  the  Type  l false  discovery  rate  for  the  large  family  of 
tests  conducted  across  all  jurisdictions  within  a table . using  the  FDR  criterion.  The  annotation 
“++"  was  used  to  indicate  a gain  or  loss  that  was  statistically  significantly  different  from  1992 
after  simultaneously  controlling  the  Type  I false  discovery  rate  for  the  number  of  tests  conducted 
across  all  jurisdictions  within  a table , using  the  FDR  criterion  In  this  second  type  of  table,  the 
annotations  and  “t”  were  also  used  as  above  to  indicate  a gain  or  loss  that  would  have  been 
significantly  different  from  1990  or  1992,  respeeti\ely,  if  only  that  jurisdiction  had  been 
examined. 

Many  of  the  tables  contain  two  or  more  types  of  statistics.  For  example,  a very  common 
table  in  the  Cross-State  Data  Compendium  contains,  for  1996.  1992.  and  1990,  the  proportion  of 
examinees  in  a given  grade  in  each  of  a particular  set  of  reporting  subgroups  (e.g..  males  and 
females,  or  each  of  the  racc/cthnicity  groups)  <Wthe  average  composite  scale  score  for  each 
subgroup.  In  a table  of  this  nature,  two  distinct  families  of  significance  tests  were  distinguished. 
The  first  family  consisted  of  the  comparisons  of  1996,  1992,  and  1990  proportions  within  each  of 
the  subgroups;  the  second  consisted  of  the  comparisons  of  1996.  1992,  and  1990  subgroup  mean 
scale  scores.  For  each  of  these  families.  Type  I error  rates  were  controlled  separately  within-row 
( for  determining  the  first  type  of  annotation)  and  simultaneously  across  juiisdictions  (toi  the 
second  type  of  annotation). 

As  a second  example,  a different  table  contained  the  percentage  of  students  reporting  that 
one  parent  had  some  college  education,  the  average  composite  scale  score  of  these  students,  and 
the  percentage  of  these  students  at  or  abo\e  each  ot  the  achievement  levels.  In  this  example, 
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three  families  of  significance  tests  were  distinguished — tests  comparing  percentages  in  the  some 
college  education  category,  tests  comparing  the  average  scale  scores  of  these  students,  and  tests 
comparing  the  percentages  exceeding  the  achievement  levels.  Again,  Type  I error  rates  were 
controlled  separately  within-row  (for  determining  the  first  type  of  annotation)  and 
simultaneously  across  jurisdictions  (for  the  second  type  of  annotation)  for  each  of  these  three 
families. 


10.5.4  Comparing  1996  Results  to  Previous  Assessments 

Since  its  inception,  one  of  NAEP’s  central  purposes  has  been  the  monitoring  of  trends  in 
achievement.  The  1996  State  Assessment  provided  an  opportunity  to  report  on  short-term  trends 
(from  1990  and/or  1992  to  1996)  in  fourth-  and  eighth-grade  mathematics  achievement  on  a 
state-by-state  basis,  as  well  as  for  the  nation  and  the  relevant  region  of  the  country.  Comparisons 
were  also  made  in  trends  of  percentages  of  students  in  specific  subgroups  across  the  years.  As  a 
result,  one  of  the  prominent  features  of  the  1996  state  report  was  the  inclusion  of  a large  number 
of  trend  comparisons  in  both  the  text  and  tables  of  the  reports  for  those  jurisdictions  that 
participated  in  the  1996  State  Assessment  and  either  the  1990  or  the  1992  Trial  State 
Assessment. 

The  samples  for  the  1990,  1992,  and  1996  State  Assessments  were  drawn  approximately 
independent  but  consisted  of  mutually  exclusive  groups  of  students,  so  that  the  sampling  plan  led 
to  a slight  negative  correlation  between  successive  years.  Although  slightly  negative,  the 
correlations  were  small,  and  therefore,  the  selections  of  text  describing  comparisons  of  1990, 
1992,  and  1996  results  were  based  on  the  types  of  significance  testing  procedures  described  in 
Section  10.5.1  and  10.5.2.  In  sections  of  the  report  where  trend  comparisons  were  carried  out  for 
a number  of  subgroups  (e.g.,  where  1996  results  were  compared  to  1992  results  for  each 
race/ethnicity  group  within  the  jurisdiction,  or  for  each  of  the  mathematics  content  strand  scales), 
the  significance  testing  procedures  incorporated  Bonferroni  adjustments,  like  those  described  in 
Section  10.5.2,  which  were  based  on  the  number  of  comparisons  being  made. 

In  addition,  a large  number  of  state  report  tables  provided  1990,  1992,  and  1996 
percentages  of  students  and  scale  score  means  for  the  subgroups  of  students  defined  by  primary 
and  secondary  reporting  variables.  In  most  of  these  tables,  three  sets  of  trend  results  were 
reported,  one  set  for  the  jurisdiction  in  question,  one  set  for  relevant  region  of  the  country,  and 
one  set  for  the  nation.  For  each  of  these  sets  of  results,  symbols  were  included  next  to  the  1996 
results  for  each  jurisdiction  indicating  which,  if  any,  of  the  reported  statistics  represented  a 
significant  change  from  the  1990  or  1992  results.  A “>”  sign  was  used  to  indicate  1996  results 
that  were  significantly  higher  than  their  corresponding  1992  levels;  a “<”  was  used  to  indicate 
1996  results  that  were  significantly  lower  than  their  corresponding  1992  levels;  and  a “«”  or 
“»”  were  placed  next  to  1996  and  1992  numbers  to  indicate  a difference  from  1990.  No  symbol 
appeared  after  results  that  did  not  differ  significantly  from  their  1990  or  1992  levels. 

As  was  done  for  text  selection,  statistical  tests  were  carried  out  using  Bonferroni 
adjustments  to  significance  levels  when  results  for  multiple  groups  were  included  in  a table.  For 
example,  in  a table  containing  1992  and  1996  mean  proficiencies  for  White,  Black,  and  Hispanic 
students,  statistical  tests  for  differences  were  carried  out  at  an  a level  of  .05/3.  It  should  be  noted 
that  national,  regional,  and  jurisdiction  comparisons  were  treated  as  separate  families  for  the 
purposes  of  obtaining  Bonferroni  adjustments.  Continuing  with  the  race/ethnicity  example, 


jurisdiction,  national,  and  regional  compaiisons  were  treated  as  three  separate  families,  each 
consisting  of  three  comparisons  and  each  of  the  required  statistical  tests  was  carried  out  at  an  cx 
level  of  .05/3. 

In  an  attempt  to  gain  greater  power,  two  separate  definitions  of  family  size  were 
employed  for  comparisons  with  large  family  sizes.  For  n levels  of  a control  variable  (e.g., 
ethnicity)  and  m levels  of  a comparison  variable  (e.g.,  number  of  hours  of  homework),  the 
standard  Bonferroni  family  size  of  n x m (m- 1 )/2  was  used.  In  addition,  when  the  m x (rn- 1 )/2 
marginal  tests  yielded  a significant  difference  for  a pair  of  categories  of  the  compatison  variable, 
the  n levels  of  the  control  variable  corresponding  to  that  pair  of  categories  were  tested  with  a 
family  size  of  n.  Significance  was  reported  if  either  definition  of  family  size  met  the  criterion. 
Interactions  were  tested  for  a mxn  table  with  t-tests  using  a family  size  n x (n- 1 ) x m x (m- 1)/4. 


10.5.5  Comparing  Proportions  Within  a Group 

Certain  analyses  in  the  state  report  involved  the  comparison  of  proportions.  One  example 
was  the  comparison  of  the  proportion  of  students  who  reported  that  a parent  graduated  from 
college  to  the  proportion  of  students  who  indicated  that  their  parents  did  not  finish  high  school  to 
determine  which  proportion  was  larger.  There  are  other  such  proportions  of  interest  in  this 
example  such  as  the  proportion  of  students  with  at  least  one  parent  graduating  from  high  school 
but  neither  parent  graduating  from  college.  For  these  types  of  analyses,  NAEP  staff  determined 
that  the  dependencies  in  the  data  could  not  be  ignored. 

Unlike  the  case  for  analyses  of  the  type  described  in  Section  10.5.1.  the  correlation 
between  the  proportion  of  students  reporting  a parent  graduated  from  college  and  the  proportion 
reporting  that  their  parents  did  not  finish  high  school  is  likely  to  be  negative  and  large.  For  a 
particular  sample  of  students,  it  is  likely  that  the  higher  the  proportion  of  students  reporting  “at 
least  one  parent  graduated  from  college”  is,  the  lower  the  proportion  of  students  reporting 
“neither  parent  graduated  from  high  school”  will  be.  A negative  dependence  will  result  in 
underestimates  of  the  standard  error  if  the  estimation  is  based  on  independence  assumptions  (as 
is  the  case  for  the  procedures  described  in  Section  10.5. 1).  Such  underestimation  can  result  in  an 
unacceptably  large  number  of  “nonsignificant"  differences  being  identified  as  significant. 

The  procedures  of  Section  10.5.1  were  modified  for  the  state  report  analyses  that 
involved  comparisons  of  proportions  within  a group.  The  modification  involved  using  a jackknife 
method  for  obtaining  the  standard  error  of  the  difference  in  dependent  proportions.  The  standard 
error  of  the  difference  in  proportions  was  obtained  by  first  obtaining  a separate  estimate  of  the 
difference  in  question  for  each  jackknife  replicate,  using  the  first  plausible  value  only,  then 
taking  the  standard  deviation  of  the  set  of  replicate  estimates  as  the  estimate.  The  procedures 
used  for  proportions  within  a group  differed  from  the  procedures  ol  Section  10.5.1  only  with 
respect  to  estimating  the  standard  error  of  the  difference;  all  other  aspects  of  the  procedures  were 
identical. 


10.5.6  Statistical  Significance  and  Estimated  Effect  Sizes 


Whenever  comparisons  were  made  between  groups,  an  attempt  was  made  to  distinguish 
between  group  differences  that  were  statistically  significant  but  rather  small  in  a practical  sense 
and  differences  that  were  both  statistically  and  practically  significant.  In  order  to  make  such 
distinctions,  a procedure  based  on  estimated  effect  sizes  was  used.  The  estimated  effect  size  for 
comparing  means  from  two  groups  was  defined  as: 


estimated  effect  size 


where  Af  refers  to  the  estimated  mean  for  group  i,  and  Sa,  refers  to  the  estimated  standard 
deviation  within  group  i.  The  within-group  estimated  standard  deviations  were  taken  to  be  the 
square  root  of  the  average  of  the  variances  of  the  set  of  five  plausible  values  taken  over  students 
for  each  imputation.  They  were  calculated  using  Westat  overall  sampling  weights. 

The  estimated  effect  size  for  comparing  proportions  was  defined  as: 

| f.  - fj  I , where/.  = 2 arcsin  ^fpit and pi  is  the  estimated  proportion  in  group  i (Cohen, 

1977). 


For  both  means  and  proportions,  no  qualifying  language  was  used  in  describing 
significant  group  differences  when  the  estimated  effect  size  exceeded  .1.  However,  when  a 
significant  difference  was  found  but  the  estimated  effect  size  was  less  than  .1,  the  qualifier 
somewhat  was  used.  For  example,  if  the  mean  scale  score  for  females  was  significantly  higher 
than  that  for  males  but  the  estimated  effect  size  of  the  difference  was  less  than  .1,  females  were 
described  as  performing  somewhat  higher  than  males. 


10.5.7  Descriptions  of  the  Magnitude  of  Percentages 

Percentages  reported  in  the  text  of  the  state  reports  are  sometimes  described  using 
quantitative  words  or  phrases.  For  example,  the  number  of  students  being  taught  by  teachers  with 
master’s  degrees  in  English  might  be  described  as  “relatively  few”  or  “almost  all,”  depending  on 
the  size  of  the  percentage  in  question.  Any  convention  for  choosing  descriptive  terms  for  the 
magnitude  of  percentages  is  to  some  degree  arbitrary.  The  rules  used  to  select  the  descriptive 
phrases  in  the  report  are  given  in  Table  10-3. 


10.5.8  Descriptions  of  Differences  Between  Differences 

When  comparing  differences  between  subgroups  across  two  assessment  years, 
descriptive  text  was  generated  as  shown  in  Table  10-4,  where  a “>”means  significantly  greater 
than  0,  a means  not  significantly  different  than  0,  and  a “>”  means  significantly  less  than  0. 
Patterns  other  than  the  six  shown  in  the  table  did  not  result  in  text  in  the  report. 


243 


nr>o 

Cue 


Table  10-3 

Rules  for  Descriptive  Terms  for  the  Magnitude  of  Percentages  Used  in  State  Reports 


Percentage 

Description  of  Text  in  Report 

p = 0 

None 

0 < p £ 8 

A small  percentage 

8 < p £ 12 

Relatively  few 

12  < p <;  18 

Less  than  one  fifth 

1 8 < p £ 22 

About  one  fifth 

22  < p <;  27 

About  one  quarter 

27  < p £ 30 

Less  than  a third 

30  < p s 36 

About  one  third 

36  < p <;  47 

Less  than  half 

47  < p $ 53 

About  half 

53  < p <;  64 

More  than  half 

64  < p s 70 

About  two  thirds 

70  < p <;  79 

About  three  quarters 

79  < p <;  89 

A large  majority 

89<p<  100 

Almost  all 

p=  100 

All 

Table  10-4 

Difference  of  Differences  ( Gaps) 


ArBl  Arffl 

> > 

> > 

> > 

< < 

< < 

< < 


(Aj-Bj)  * (A2-B2)  Text  

> The  gap  between  A and  B is  larger  in  1996  than  in  1992. 

= The  gap  between  A and  B is  not  significantly  different  in  1996 

than  in  1992. 

< The  gap  between  A and  B is  smaller  in  1996  than  in  1992. 

> The  gap  between  A and  B is  smaller  in  1996  than  in  1992. 

= The  gap  between  A and  B is  not  significantly  different  in  1996 

than  in  1992. 

< The  gap  between  A and  B is  larger  in  1996  than  in  1992. 


lAi  and  S,  are  the  majority  and  largest  minority  subgroups  in  1996. 
2A2  and  B2  arc  the  corresponding  subgroups  in  1992. 
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SUMMARY  OF  PARTICIPATION  RATES 

Guidelines  for  Sample  Participation  and 
Explanation  of  the  Derivation  of  Weighted  Participation  Rates 
for  the  1996  State  Assessment  Program  in  Mathematics 

Keith  F.  Rust 
Westat,  Inc . 

and 

Eugene  G.  Johnson  and  Nada  Ballator 
Educational  Testing  Services 


Introduction 

Since  1989,  state  representatives,  the  National  Assessment  Governing  Board  (NAGB), 
several  committees  of  advisors  external  to  the  National  Assessment  of  Educational  Progress 
(NAEP),  and  the  National  Center  for  Education  Statistics  (NCES)  have  engaged  in  numerous 
discussions  about  the  procedures  for  reporting  the  NAEP  state  assessment  results.  From  the 
outset  of  thes  i discussions,  it  was  recognized  that  sample  participation  rates  across  the 
jurisdictions  have  to  be  uniformly  high  to  permit  fair  and  valid  comparisons.  Therefore,  NCES 
established  guidelines  for  school  and  student  participation  for  the  first  two  Trial  State 
Assessment  programs  in  1990  and  1992. 

The  1994  Trial  State  Assessment  program  used  an  expanded  set  of  participation 
guidelines.  The  guidelines  were  expanded  in  two  ways.  First,  new  guidelines  were  designed  to 
preempt  publication  of  results  from  jurisdictions  for  which  participation  rates  were  low  enough 
to  suggest  the  possibility  of  appreciable  nonresponse  bias.  The  new  guidelines  were  congruent 
both  with  NAGB  policies  as  well  as  the  resolutions  of  the  Education  Information  Advisory 
Committee  (EIAC).  Second,  existing  guidelines  were  extended  to  cover  the  presence  of  separate 
public  and  nonpublic  school  samples  in  the  1994  Trial  State  Assessment. 

For  the  NAEP  1996  State  Assessment  program,  the  participation  guidelines  implemented 
in  1994  were  again  applied.  This  appendix  provides: 

• Participation  rate  information  for  the  NAEP  1996  State  Assessment  of 
mathematics  at  grades  4 and  8 for  both  public  and  nonpublic  school 
samples.  This  information  will  also  appear  in  appendices  in  the  NAEP  1996 
Mathematics  Report  Card  and  the  NAEP  1996  Mathematics  State  Report. 

• An  explanation  of  the  guidelines  and  notations  used  in  1996.  In  brief,  the 
guidelines  cover  levels  of  school  and  student  participation,  both  overall  and 
for  particular  population  classes,  separately  for  both  public  and  nonpublic 
school  samples.  Consistent  with  the  NCES  standards,  weighted  data  are  used 
to  calculate  all  participation  rales  for  sample  surveys,  and  weighted  rates  are 
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provided  in  the  reports.  The  procedures  used  to  derive  the  weighted  school 
and  student  participation  rates  are  provided  immediately  after  the  discussion 
of  the  guidelines  and  notations. 

• A set  of  tables  that  provides  the  1996  participation  rate  information  for 
the  NAEP  1996  State  Assessment  of  Mathematics.  Separate  information  is 
provided  for  the  public  and  nonpublic  school  samples.  The  sample  for 
nonpublic  schools  includes  schools  not  directed  by  traditional  local  or  state 
government  agencies,  such  as  those  administered  by  Cathoiic  dioceses,  other 
religious  and  nonsectarian  schools,  schools  administered  by  the  Bureau  of 
Indian  Affairs  (BIA),  and  schools  administered  by  the  Department  of 
Defense  in  the  United  States.  Because  the  aggregate  sample  across  all 
participating  jurisdictions  (public  or  nonpublic)  is  not  necessarily 
representative  of  the  nation,  the  weighted  participation  rates  across 
participating  jurisdictions  have  not  been  analyzed.  However,  the  counts  from 
the  national  assessment  have  been  included  to  provide  some  context  for 
interpreting  the  summary  of  activities  in  each  individual  state  and  territory 
and  for  each  type  of  school.  Please  note  that  in  the  NAEP  1996  state 
assessment.  Department  of  Defense  Domestic  Dependent  Elementary  and 
Secondary  (DDESS)  and  Department  of  Defense  Dependents  Schools 
(DoDDS)  schools  were  included  as  two  separate  jurisdictions;  for  this  report 
and  any  future  reports  including  1996  State  Assessment  data,  the  two 
Department  of  Defense  Education  Activity  (DoDEA)  jurisdictions  will  be 
reported  as  jurisdictions  having  only  public  schools. 


Notations  for  Use  in  Reporting  School  and  Student  Participation  Rates 

Unless  the  overall  participation  rate  is  sufficiently  high  for  a jurisdiction,  there  is  a risk 
that  the  assessment  results  for  that  jurisdiction  are  subject  to  appreciable  nonresponse  bias. 
Moreover,  even  if  the  overall  participation  rate  is  high,  there  may  be  significant  nonresponse  bias 
if  the  nonparticipation  that  does  occur  is  heavily  concentrated  among  certain  types  of  schools  or 
students.  The  following  guidelines  concerning  school  and  student  participation  rates  in  the  NAEP 
state  assessment  program  were  established  to  address  four  significant  ways  in  which  nonresponse 
bias  could  be  introduced  into  the  jurisdiction  sample  estimates.  The  conditions  that  will  result  in 
the  publication  of  a jurisdiction's  results  are  presented  below.  Also  presented  below  are  the 
conditions  that  will  result  in  a jurisdiction  receiving  a notation  in  the  1996  reports.  Note  that  in 
order  for  a jurisdiction's  results  to  be  published  with  no  notations,  that  jurisdiction  must  satisfy 
all  guidelines.  The  guidelines  are  applied  separately  to  the  fourth-  and  eighth-grade  mathematics 
samples. 


Guidelines  on  the  Publication  of  NAEP  Results 

Guideline  I - Publication  of  Public  School  Results.  A jurisdiction  will  have  its  public 
school  results  published  in  the  IWb  NAEP  Mathematics  Report  Card  (or  in  other  reports  that 
include  all  state-level  results)  if  and  only  if  its  weighted  participation  rate  for  the  initial  sample 
of  public  schools  is  greater  than  or  equal  to  70  percent.  Similarly,  a jurisdiction  will  receive  a 


separate  NAEP  1996  Mathematics  State  Report  if  and  only  if  its  weighted  participation  rate  for 
the  initial  sample  of  public  schools  is  greater  than  or  equal  to  70  percent. 

Guideline  2 - Publication  of  Nor  public  School  Results.  A jurisdiction  will  have  its 
nonpublic  school  results  published  in  the  1996  NAEP  Mathematics  Report  Card  (or  in  other 
reports  that  include  all  state-level  results)  if  and  only  if  its  weighted  participation  rate  for  the 
initial  sample  of  nonpublic  schools  is  greater  than  or  equal  to  70  percent  and  meets  minimum 
sample  size  requirements.1  A jurisdiction  eligible  to  receive  a separate  NAEP  1996  Mathematics 
State  Report  under  Guideline  1 will  have  its  nonpublic  school  results  included  in  that  report  if 
and  only  if  that  jurisdiction’s  weighted  participation  rate  for  the  initial  sample  of  nonpublic 
schools  is  greater  than  or  equal  to  70  percent  and  meets  minimum  sample  size  requirements.  If  a 
jurisdiction  meets  Guideline  2 but  fails  to  meet  Guideline  1,  a separate  State  Report  will  be 
produced  containing  only  nonpublic  school  results. 

Guideline  3 - Publication  of  Combined  Public  and  Nonpublic  School  Results . A 
jurisdiction  will  have  its  combined  results  published  in  the  1996  NAEP  Mathematics  Report 
Card  (or  in  other  reports  that  include  all  state-level  results)  if  and  only  if  both  Guidelines  1 and  2 
are  satisfied.  Similarly,  a jurisdiction  eligible  to  receive  a separate  NAEP  1996  Mathematics 
State  Report  under  Guideline  1 will  have  its  combined  results  included  in  that  report  if  and  only 
if  Guideline  2 is  also  met. 

Discussion.  If  a jurisdiction’s  public  or  nonpublic  school  participation  rate  for  the  initial 
sample  of  schools  is  below  70  percent  there  is  a substantial  possibility  that  bias  will  be 
introduced  into  the  assessment  results.  This  possibility  remains  even  after  making  statistical 
adjustments  to  compensate  for  school  nonparticipation.  There  remains  the  likelihood  that,  in 
aggregate,  the  substitute  schools  are  sufficiently  dissimilar  from  the  originals  that  they  are 
replacing  and  represent  too  great  a proportion  of  the  population  to  discount  such  a difference. 
Similarly,  the  assumptions  underlying  the  use  of  statistical  adjustments  to  compensate  for 
nonparticipation  are  likely  to  be  significantly  violated  if  the  initial  response  rate  falls  below  the 
70  percent  level.  Guidelines  1,  2,  and  3 take  this  into  consideration.  These  guidelines  are 
congruent  with  current  NAGB  policy,  which  requires  that  data  for  jurisdictions  that  do  not  have  a 
70  percent  before-substitution  participation  rate  be  reported  "in  a different  format,"  and  with  the 
Education  Information  Advisory  Committee  (EIAC)  resolution,  which  calls  for  data  from  such 
jurisdictions  not  to  be  published. 

Guidelines  on  Notations  of  NAEP  Results 

Guideline  4 - Notation  for  Overall  Public  School  Participation  Rate.  A jurisdiction  that 
meets  Guideline  1 will  receive  a notation  if  its  weighted  participation  rate  for  the  initial  sample 
of  public  schools  was  below  85  percent  and  the  weighted  public  school  participation  rate  after 
substitution  was  below  90  percent. 


‘Minimum  sample  size  requirements  for  reporting  nonpublic  school  data  consist  of  two  components:  {Da  school  sample 
size  of  six  or  more  participating  schools  and  (2)  an  assessed  student  sample  size  of  at  least  62. 
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Guideline  5 - Notation  for  Overall  Nonpublic  School  Participation  Rate.  A jurisdiction 
that  meets  Guideline  2 will  receive  a notation  if  its  weighted  participation  rate  for  the  initial 
sample  of  nonpublic  schools  was  below  85  percent  and  the  weighted  nonpublic  school 
participation  rate  after  substitution  was  below  90  percent. 

Discussion.  For  jurisdictions  that  did  not  use  substitute  schools,  the  participation  rates 
are  based  on  participating  schools  from  the  original  sample.  In  these  situations,  the  NCES 
standards  specify  weighted  school  participation  rates  of  at  least  85  percent  to  guard  against 
potential  bias  due  to  school  nonresponse.  Thus  the  first  part  of  these  guidelines,  referring  to  the 
weighted  school  participation  rate  for  the  initial  sample  of  schools,  is  in  direct  accordance  with 
NCES  standards. 

To  help  ensure  adequate  sample  representation  for  each  jurisdiction  participating  in  the 
NAEP  1996  state  assessments,  NAEP  provided  substitutes  for  nonparticipating  public  and 
nonpublic  schools.  For  jurisdictions  that  used  substitute  schools,  the  assessment  results  will  be 
based  on  the  student  data  from  all  schools  participating  from  both  the  original  sample  and  the  list 
of  substitutes  (unless  both  an  initial  school  and  its  substitute  eventually  participated,  in  which 
case  only  the  data  from  the  initial  school  will  be  used). 

The  NCES  standards  do  not  explicitly  address  the  use  of  substitute  schools  to  replace 
initially  selected  schools  that  decide  not  to  participate  in  the  assessment.  However,  considerable 
technical  consideration  was  given  to  this  issue.  Even  though  the  characteristics  of  the  substitute 
schools  were  matched  as  closely  as  possible  to  the  characteristics  of  the  initially  selected  schools, 
substitution  does  not  entirely  eliminate  bias  due  to  the  nonparticipation  of  initially  selected 
schools.  Thus,  for  the  weighted  school  participation  rates  including  substitute  schools,  the 
guidelines  were  set  at  90  percent. 

If  a jurisdiction  meets  either  standard  (i.e.,  85  percent  or  higher  prior  to  substitution  or 
90  percent  or  higher  after  substitution),  there  will  be  no  notation  for  the  relevant  overall  school 
participation  rate. 

Guideline  6 - Notation  for  Strata-Specific  Public  School  Participation  Rates.  A 
jurisdiction  that  is  not  already  receiving  a notation  under  Guideline  4 will  receive  a notation  if 
the  sample  of  public  schools  included  a class  of  schools  with  similar  characteristics  that  had  a 
weighted  participation  rate  (after  substitution)  of  below  80  percent,  and  from  which  the 
nonparticipating  schools  together  accounted  for  more  than  five  percent  of  the  jurisdiction's  total 
weighted  sample  of  public  schools.  The  classes  of  schools  from  each  of  which  a jurisdiction 
needed  minimum  school  participation  levels  were  determined  by  degree  cf  urbanization,  minority 
enrollment,  and  median  household  income  of  the  area  in  which  the  school  is  located. 

Guideline  7 - Notation  for  Strata-Specific  Nonpublic  School  Participation  Rates.  A 
jurisdiction  that  is  not  already  receiving  a notation  under  Guideline  5 will  receive  a notation  if 
the  sample  of  nonpublic  schools  included  a class  of  schools  with  similar  characteristics  that  had  a 
weighted  participation  rate  (after  substitution)  of  below  80  percent,  and  from  which  the 
nonparticipating  schools  together  accounted  for  more  than  five  percent  of  the  jurisdiction’s  total 
weighted  sample  of  nonpublic  schools.  The  classes  of  schools  from  each  of  which  a jurisdiction 
needed  minimum  school  participation  levels  w'erc  determined  by  type  of  nonpublic  school 
(Catholic  versus  non-Catholic)  and  location  (metropolitan  versus  nonmetropolitan). 
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Discussion.  The  NCES  standards  specify  that  attention  should  be  given  to  the 
representativeness  of  the  sample  coverage.  Thus,  if  some  important  segment  of  the  jurisdiction’s 
population  is  not  adequately  represented,  it  is  of  concern,  regardless  of  the  overall  participation 
rate. 


If  nonparticipating  schools  are  concentrated  within  a particular  class  of  schools,  the 
potential  for  substantial  bias  remains,  even  if  the  overall  level  of  school  participation  appears  to 
be  satisfactory.  Nonresponse  adjustment  cells  for  public  schools  have  been  formed  within  each 
jurisdiction,  and  the  schools  within  each  cell  are  similar  with  respect  to  minority  enrollment, 
degree  of  urbanization,  and/or  median  household  income,  as  appropriate  for  each  jurisdiction. 
For  nonpublic  schools,  nonresponse  adjustment  cells  are  determined  by  type  and  location  of 
school. 


If  the  weighted  response  rate,  after  substitution,  for  a single  adjustment  cell  falls  below 
80  percent,  and  more  than  five  percent  (weighted)  of  the  sampled  schools  are  nonparticipants 
from  such  a cell,  the  potential  for  nonresponse  bias  is  too  great.  These  guidelines  are  based  on 
the  NCES  standard  for  stratum-specific  school  response  rates.  This  criterion,  that  the 
nonparticipating  schools  in  a class  constitute  more  than  five  percent  of  the  jurisdiction’s 
population  (i.e.,  the  total  weighted  samples  of  public  or  nonpublic  schools),  is  included  to  insure 
that  the  notation  is  triggered  only  if  the  subgroup  having  a response  below  80  percent  makes  up 
at  least  one  quarter  of  the  jurisdiction’s  student  population  (separately  by  public  and  nonpublic 
schools).  This  means  that  a notation  is  triggered  only  in  cases  where  a substantial  subgroup  has 
experienced  an  unsatisfactory  response.  Without  this  criterion,  it  is  possible  that  a response  rate 
of  just  below  80  percent  in  a small  population  subgroup  could  trigger  a notation  inappropriately. 
These  guidelines  are  based  on  the  NCES  standard  for  stratum-specific  school  response  rates. 

Guideline  8 - Notation  for  Overall  Student  Participation  Rate  in  Public  Schools.  A 
jurisdiction  that  meets  Guideline  1 will  receive  a notation  if  the  weighted  student  response  rate 
within  participating  public  schools  was  below  85  percent. 

Guideline  9 - Notation  for  Overall  Student  Participation  Rate  in  Nonpublic  Schools.  A 
jurisdiction  that  meets  Guideline  2 will  receive  a notation  if  the  weighted  student  response  rate 
within  participating  nonpublic  schools  was  below  85  percent. 

Discussion.  These  guidelines  follow  the  NCES  standard  of  85  percent  for  overall  student 
participation  rates.  The  weighted  student  participation  rate  is  based  on  all  eligible  students  from 
initially  selected  or  substitute  schools  who  participated  in  the  assessment  in  either  an  initial 
session  or  a make-up  session.  If  the  rate  falls  below  85  percent,  the  potential  for  bias  due  to 
students'  nonresponse  is  too  great. 

Guideline  10  - Notation  for  Strata-Speciftc  Student  Participation  Rates  in  Public 
Schools.  A jurisdiction  that  is  not  already  receiving  a notation  under  Guideline  8 will  receive  a 
notation  if  the  sampled  students  within  participating  public  schools  included  a class  of  students 
with  similar  characteristics  that  had  a weighted  student  response  rate  of  below  80  percent,  and 
from  which  the  nonresponding  students  together  accounted  for  more  than  five  percent  of  the 
jurisdiction's  weighted  assessable  public  school  student  sample.  Student  groups  from  which  a 
jurisdiction  needed  minimum  levels  of  participation  were  determined  by  the  age  of  the  student, 
whether  or  not  the  student  was  classitied  as  a student  with  a disability  (St))  or  of  limited  English 
proficiency  (LEP),  and  the  type  of  assessment  session  (monitored  or  unmonitored),  as  well  as 
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school  level  of  urbanization,  minority  enrollment,  and  median  household  income  of  the  area  in 
which  the  school  is  located. 

Guideline  II  - Notation  for  Strata-Specific  Student  Participation  Rates  in  Nonpublic 
Schools.  A jurisdiction  that  is  not  already  receiving  a notation  under  Guideline  9 will  receive  a 
notation  if  the  sampled  students  within  participating  nonpublic  schools  included  a class  of 
students  with  similar  characteristics  that  had  a weighted  student  response  rate  of  below  80 
percent,  and  from  which  the  nonresponding  students  together  accounted  for  more  than  five 
percent  of  the  jurisdiction’s  weighted  assessable  nonpublic  school  student  sample.  Student 
groups  from  which  a jurisdiction  needed  minimum  levels  of  participation  were  determined  by  the 
age  of  the  student,  whether  or  not  the  student  was  classified  as  a student  with  a disability  (SD)  or 
of  limited  English  proficiency  (LEP),  and  the  type  of  assessment  session  (monitored  or 
unmonitored),  as  well  as  type  and  location  of  school. 

Discussion.  These  guidelines  address  the  fact  that  if  nonparticipating  students  are 
concentrated  within  a particular  class  of  students,  the  potential  for  substantial  bias  remains,  even 
if  the  overall  student  participation  level  appears  to  be  satisfactory.  Student  nonresponse 
adjustment  cells  have  been  formed  using  the  school-level  nonresponse  adjustment  cells,  together 
with  the  student’s  age  and  the  nature  of  the  assessment  session  (unmonitored  or  monitored). 

If  the  weighted  response  rate  for  a single  adjustment  cell  falls  below  80  percent,  and 
more  than  five  percent  (weighted)  of  the  invited  students  who  do  not  participate  in  the 
assessment  are  from  such  a cell,  the  potential  for  nonresponse  bias  is  too  great.  These  guidelines 
are  based  on  the  NCES  standard  for  stratum-specific  student  response  rates. 


Derivation  of  Weighted  Participation  Rates 

Weighted  School  Participation  Rates.  The  weighted  school  participation  rates  within 
each  jurisdiction  provide  the  percentages  of  fourth-  or  eighth-grade  students  in  public  (or 
nonpublic)  schools  who  are  represented  by  the  schools  participating  in  the  assessment,  prior  to 
statistical  adjustments  for  school  nonresponse. 

Two  sets  of  weighted  school  participation  rates  are  computed  for  each  jurisdiction,  one 
for  public  schools  and  one  for  nonpublic  schools.  Each  set  consists  of  two  weighted  participation 
rates.  The  first  is  the  weighted  participation  rate  for  the  initial  sample  of  schools.  This  rate  is 
based  only  on  those  schools  that  were  initially  selected  for  the  assessment.  The  numerator  of  this 
rate  is  the  sum  of  the  number  of  students  represented  by  each  initially  selected  school  that 
participated  in  the  assessment.  The  denominator  is  the  sum  of  the  number  of  students  represented 
by  each  of  the  initially  selected  schools  found  to  have  eligible  students  enrolled.  This  includes 
both  participating  and  nonparticipating  schools. 

The  second  is  the  weighted  participation  rate  after  substitution.  The  numerator  of  this 
rate  is  the  sum  of  the  number  of  students  represented  by  each  of  the  participating  schools, 
whether  originally  selected  or  a substitute.  The  denominator  is  the  same  as  that  for  the  weighted 
participation  rate  for  the  initial  sample.  This  means  that,  for  a given  jurisdiction  and  type  of 
school,  the  weighted  participation  rate  after  substitution  is  always  at  least  as  great  as  the 
weighted  participation  rate  for  the  initial  sample  of  schools. 


In  general,  different  schools  in  the  sample  can  represent  different  numbers  of  students  in 
the  jurisdiction's  population.  The  number  of  students  represented  by  an  initially  selected  school 
(the  school  weight)  is  the  fourth-  or  eighth-grade  enrollment  of  the  school  divided  by  the 
probability  that  the  school  was  included  in  the  sample.  For  instance,  a selected  school  with  a 
eighth-grade  enrollment  of  150  and  a selection  probability  of  0.2  represents  750  students  from 
that  jurisdiction.  The  number  of  students  represented  by  a substitute  school  is  the  number  of 
students  represented  by  the  replaced  nonparticipating  school. 

Because  each  selected  school  represents  different  numbers  of  students  in  the  population, 
the  weighted  school  participation  rates  may  differ  somewhat  from  the  simple  unweighted  rates. 
(The  unweighted  rates  are  calculated  from  the  counts  of  schools  by  dividing  the  number  of 
participating  schools  by  the  number  of  schools  in  the  sample  with  eligible  students  enrolled.)  The 
difference  between  the  weighted  and  the  unweighted  rates  is  potentially  largest  in  smaller 
jurisdictions  where  all  schools  with  fourth-  or  eighth-grade  students  were  included  in  the  sample 
(that  is,  where  no  substitutes  are  available).  In  those  jurisdictions,  each  school  represents  only  its 
own  students.  Therefore,  the  nonparticipation  of  a large  school  reduces  the  weighted  school 
participation  rate  by  a greater  amount  than  does  the  nonparticipation  of  a small  school. 

The  nonparticipation  of  larger  schools  also  has  greater  impact  than  that  of  smaller 
schools  on  reducing  weighted  school  participation  rates  in  larger  jurisdictions  where  fewer  than 
all  of  the  schools  were  included  in  the  sample.  However,  since  the  number  of  students 
represented  by  each  school  is  more  nearly  constant  in  larger  states,  the  difference  between  the 
impact  of  nonparticipation  by  either  large  or  small  schools  is  less  marked  than  in  jurisdictions 
where  all  schools  were  selected. 

In  general,  the  greater  the  population  in  the  jurisdiction,  the  smaller  the  difference 
between  the  weighted  and  unweighted  school  participation  rates.  However,  even  in  the  less 
populous  jurisdictions,  the  differences  tend  to  be  small. 

Weighted  Student  Participation  Rate.  The  weighted  student  participation  rate  provides 
the  percentage  of  the  eligible  student  population  from  participating  schools  within  the 
jurisdiction  that  are  represented  by  the  students  who  participated  in  the  assessment  (in  either  an 
initial  session  or  a make-up  session).  Separate  weighted  student  participation  rates  were 
calculated  for  public  and  nonpublic  school  students.  The  eligible  student  population  from 
participating  schools  (public  or  nonpublic)  within  a jurisdiction  consists  ot  all  students  who  were 
in  the  fourth  or  eighth  grade,  who  attended  a school  that,  if  selected,  would  have  participated  and 
who.  if  selected,  would  not  have  been  excluded  from  the  assessment.  The  numerator  of  this  rate 
is  the  sum,  across  all  assessed  students,  of  the  number  of  students  represented  by  each  assessed 
student  (prior  to  adjustment  for  student  nonparticipation).  The  denominator  is  the  sum  of  the 
number  of  students  represented  by  each  selected  student  who  was  invited  and  eligible  to 
participate  (i.e..  not  excluded),  including  students  who  did  not  participate.  Thus,  the  denominator 
is  an  estimate  of  the  total  number  of  assessable  students  in  the  group  ol  schools  within  the 
jurisdiction  that  would  have  participated  if  selected. 

The  number  of  students  represented  by  a single  selected  student  (the  student  weight)  is 
1 .0  divided  by  the  overall  probability  that  (he  student  was  selected  for  assessment.  In  general,  the 
number  of  students  from  a jurisdiction’s  population  represented  by  it  sampled  student  is 
approximately  constant  across  students.  Consequently,  there  is  little  difference  between  the 
weighted  student  participation  rate  and  the  unweighted  student  participation  rate. 


Weighted  Overall  School  and  Student  Participation  Rate.  An  overall  indicator  of  the 
effect  of  nonparticipation  by  both  students  and  schools  is  given  by  the  overall  participation  rate. 
Separate  overall  rates  were  calculated  for  public  and  nonpublic  school  samples.  For  each  school 
type  (public  or  nonpublic),  these  weights  were  calculated  as  the  product  of  the  weighted  school 
participation  rate  (after  substitution),  and  the  weighted  student  participation  rate.  For 
jurisdictions  having  a high  overall  participation  rate,  the  potential  is  low  for  bias  to  be  introduced 
through  either  school  nonparticipation  or  student  nonparticipation.  This  rate  provides  a summary 
measure  that  indicates  the  proportion  of  the  jurisdiction's  fourth-  or  eighth-grade  public  or 
nonpublic  school  student  population  that  is  directly  represented  by  the  final  student  sample. 
When  the  overall  rate  is  high,  the  adjustments  for  nonresponse  that  are  used  in  deriving  the  final 
survey  weights  are  likely  to  be  effective  in  maintaining  nonresponse  bias  at  a negligible  level. 
Conversely,  when  the  overall  rate  is  relatively  low  there  is  a greater  chance  that  a non-negligible 
bias  remains  even  after  making  such  adjustments. 

The  overall  rate  is  not  used  in  establishing  the  guidelines/notations  for  school  and 
student  participation,  since  guidelines  already  exist  covering  school  and  student  participation 
separately . 


Derivation  of  Weighted  Percentages  for  Excluded  Students 

Weighted  Percentage  of  Excluded  Students.  The  weighted  percentage  of  excluded 
students  estimates  the  percentage  of  the  fourth-  or  eighth-grade  population  in  the  jurisdiction’s 
schools  that  is  represented  by  the  students  who  were  excluded  from  the  assessment,  after 
accounting  for  school  nonparticipation.  The  numerator  is  the  sum,  across  all  excluded  students, 
of  the  number  of  students  represented  by  each  excluded  student.  The  denominator  is  the  sum  of 
the  number  of  students  represented  by  each  of  the  students  who  was  sampled  (and  had  not 
withdrawn  from  the  school  at  the  time  of  the  assessment). 

Weighted  Percentage  of  Students  with  Disabilities  (SD).  The  weighted  percentage  of  SD 
students  estimates  the  percentage  of  the  fourth-  or  eighth-grade  population  in  the  jurisdiction’s 
schools  represented  by  the  students  who  were  classified  as  SD,  after  accounting  for  school 
nonparticipation.  The  numerator  is  the  sum,  across  all  students  classified  as  SD,  of  the  number  of 
students  represented  by  each  SD  student.  The  denominator  is  the  sum  of  the  number  of  students 
represented  by  each  of  the  students  who  was  sampled  (and  had  not  withdrawn  from  the  school  at 
the  time  of  the  assessment). 

Weighted  Percentage  of  Excluded  SD  Students.  The  weighted  percentage  of  excluded  SD 
students  estimates  the  percentage  of  students  in  the  jurisdiction  who  are  represented  by  those  SD 
students  excluded  from  the  assessment,  after  accounting  for  school  nonparticipation.  The 
numerator  is  the  sum,  across  all  students  classified  as  SD  and  excluded  from  the  assessment,  of 
the  number  of  students  represented  by  each  excluded  SD  student.  The  denominator  is  the  sum  of 
the  number  of  students  represented  by  each  of  the  students  who  was  sampled  (and  had  not 
withdrawn  from  the  school  at  the  time  of  the  assessment). 

Weighted  Percentage  of  Limited  English  Pro  ficiency  (LEP)  Students.  The  weighted 
percentage  of  LEP  students  estimates  the  percentage  of  the  fourth-  or  eighth-grade  population  in 
the  jurisdiction’s  schools  represented  by  the  students  who  were  classified  as  LEP,  after 
accounting  for  school  nonparticipation.  The  numerator  is  the  sum,  across  all  students  classified 
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as  LEP,  of  the  number  of  students  represented  by  each  LEP  student.  The  denominator  is  the  sum 
of  the  number  of  students  represented  by  each  of  the  students  who  was  sampled  (and  had  not 
withdrawn  from  the  school  at  the  time  of  the  assessment). 

Weighted  Percentage  of  Excluded  LEP  Students.  The  weighted  percentage  of  LEP 
students  who  were  excluded  estimates  the  percentage  of  students  in  the  jurisdiction  represented 
by  those  LEP  students  excluded  from  the  assessment,  after  accounting  for  school 
nonparticipation.  The  numerator  is  the  sum,  across  all  students  classified  as  LEP  and  excluded 
from  the  assessment,  of  the  number  of  students  represented  by  each  excluded  LEP  student.  The 
denominator  is  the  sum  of  the  number  of  students  represented  by  each  student  who  was  sampled 
(and  had  not  withdrawn  from  the  school  at  the  time  of  the  assessment). 

Note:  All  percentages  are  based  on  student  weights  that  have  been  adjusted  for  school-level 
nonresponse.  All  weighted  percentages  were  calculated  separately  for  public  and  nonpublic 
school  samples. 
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Table  B-l 

School  Participation  Rates,  Grade  4,  1996  Mathematics  Assessment,  Public  Schools 
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Siv  preceding  »c\t  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the  derivation  of  weighted 
participation.  For  Delaware,  the  District  of  Columbia.  DDESS.  and  Guam,  the  State  Assessment  was  based  on  all  eligible  public 
vchook  Me,  there  was  no  sampling  of  public  schools) 

*1  he  state  s public  school  weighted  participation  rate  for  the  initial  sample  of  schools  w as  below  85Cf  and  the  weighted  school 
participation  rate  after  substitution  was  below  . 

fhe  nonparticipating  public  schools  included  a class  of  schools  with  similar  characteristics,  which  together  accounted  for  more  than 
5*#  ol  the  state's  total  fourth-grade  weighted  sample  of  public  schools. 

In  Colorado.  Georgia.  Iowa,  and  New  Jersey,  the  materials  from  one  school  that  conducted  an  assessment  were  lost  in  shipping.  The 
school  is  included  in  the  counts  of  participating  schools,  both  before  and  after  substitution.  However,  in  the  weighted  results,  the 
school  is  treated  in  the  same  manner  as  a nonparticipating  school  because  no  student  responses  were  available  for  analysis  and 
i c porting 

SOCRCK  National  Center  for  F.ducatmn  Statistics.  National  Assessment  ol  Educational  Progress  (NAEP).  19%  Mathematics 
Assessment 
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Table  B-2 

School  Participation  Rates,  Grade  4,  1996  Mathematics  Assessment,  Nonpublic  Schools 


Nonpublic 

Schools 

Weighted 

Percentage 

School 

Participation 

Before 

Substitution 

Weighted 

Percentage 

School 

Participation 

After 

Substitution 

Number  of 
Schools  in 
Original 
Sample 

Number  of 
Schools  not 
Eligible 

Number  of 
Schools  in 
Original 
Sample  that 
Participated 

Number  of 
Substituted 
Schools 
Provided 

Number  of 
Substituted 
Schools  that 
Participated 

Total  Number 
of  Schools 
that 

Participated 

NATION 

79 

79 

105 

5 

77 

0 

0 

77 

JURISDICTIONS 

A*abamo' 

72 

72 

15 

0 

10 

5 

0 

10 

Ari2ona: 

78 

87 

13 

1 

9 

1 

1 

10 

Arkansas 

86 

86 

12 

2 

8 

1 

0 

8 

Coiiforn>o5 

73 

73 

19 

4 

11 

4 

0 

11 

Colorado* 

76 

76 

15 

2 

10 

3 

0 

10 

Connecticut* 

75 

75 

19 

2 

13 

4 

0 

13 

De'aware1 

.11 

43 

44 

11 

12 

9 

1 

13 

District  of  Columbia1 

63 

66 

34 

5 

17 

4 

1 

1 

18 

Florida1 

66 

73 

19 

3 

11 

4 

12 

Georgia 

99 

99 

14 

0 

13 

1 

0 

13 

Guam' 

78 

78 

14 

1 

9 

0 

0 

9 

Indiana' 

79 

86 

24 

5 

14 

5 

1 

15 

Iowa’ 

82 

82 

19 

0 

15 

3 

c 

15 

Kentucky 

87 

87 

19 

3 

13 

3 

0 

13 

Louisiana 

86 

86 

24 

2 

19 

3 

0 

19 

Maine* 

71 

74 

16 

5 

7 

4 

1 

8 

Maryland1 

57 

57 

24 

5 

ii 

7 

0 

11 

Massachusetts' 

84 

84 

19 

i 

15 

2 

0 

15 

Michigan 

86 

94 

22 

3 

16 

2 

2 

18 

Minnesota' 

78 

73 

23 

4 

15 

4 

0 

15 

Mississippi' 

79 

79 

15 

0 

n 

4 

0 

1 1 

Missouri 

99 

TOO 

23 

0 

22 

1 

1 

23 

Montana 

94 

94 

19 

6 

10 

3 

0 

10 

Nebraska 

91 

91 

27 

2 

22 

3 

0 

2? 

Nevada 

9) 

100 

11 

2 

8 

1 

1 

9 

New  Jersey' 

64 

75 

26 

4 

14 

8 

2 

16 

New  Mexico 

90 

90 

20 

5 

13 

1 

0 

13 

New  York' 

83 

91 

27 

2 

21 

4 

2 

23 

North  Dakota' 

68 

68 

21 

2 

12 

7 

0 

12 

Oregon’ 

34 

34 

15 

3 

4 

7 

0 

4 

Pennsylvania’ 

66 

66 

35 

1 

19 

15 

0 

19 

Texas’ 

64 

64 

10 

4 

4 

2 

0 

4 

Utah' 

81 

81 

10 

1 

7 

2 

0 

7 

Vermont' 

74 

74 

19 

6 

9 

3 

0 

9 

Wisconsin' 

68 

73 

38 

3 

23 

12 

2 

25 

Wyoming 

82 

95 

11 

3 

6 

2 

1 

7 

See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for 
the  derivation  of  weighted  participation.  For  Guam,  the  State  Assessment  was  based  on  all  eligible  schools 
(i.e.,  there  was  no  sampling  of  nonpublic  schools). 

2The  state's  nonpublic  school  weighted  participation  rate  for  the  initial  sample  was  less  than  70%-. 

'The  state’s  nonpublic  school  weighted  participation  rate  for  the  initial  sample  of  schools  was  below  85% 
and  the  weighted  school  participation  rate  after  substitution  was  below  90%. 

7The  nonparticipating  nonpublic  schools  included  a class  of  schools  with  similar  characteristics,  which 
together  accounted  for  more  than  5%  of  the  state’s  total  fourth-grade  weighted  sample  of  nonpublic  schools. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP), 
1996  Mathematics  Assessment. 
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Table  B-3 

Student  Participation  Rates , Grade  4,  1996  Mathematics  Assessment,  Public  Schools 


Public 

Schools 

Weighted 

Percentage 

Student 

PorKdpcilkm 

After 

Make-Ups 

Number  of 
Students  in 
Original 
Sample 

Number  of 
Student*  m 
Supplemental 
Somple 

Number  of 
Student* 
Withdraw 

Number  of 
Student* 
Excluded 

Number  of 
Students 
To  Be 
Asteued 

Number  of 
Student* 
Assessed 
initial 
Session* 

Number  of 
Students 

Assessed 

Makeups 

Total  Number 
of  Students 

Assessed 

NATION 

95 

5.756 

68 

197 

5.493 

5.148 

67 

5.215 

JURISDICTIONS 

AloDomo 

96 

2.739 

72 

35 

80 

2 646 

2 479 

62 

2541 

Alosko 

91 

2.634 

38 

83 

56 

2 528 

2.299 

5 

2304 

Afzono 

95 

2.467 

87 

132 

182 

2.240 

2095 

18 

2113 

Attars 

96 

2237 

58 

93 

72 

2 130 

2039 

8 

2047 

Catfcma 

94 

2464 

41 

73 

245 

2.187 

2060 

3 

2063 

Co'oraao 

95 

2923 

92 

124 

133 

2753 

2599 

10 

2609 

Connecticut 

96 

2653 

53 

99 

123 

2.684 

2560 

5 

2565 

Delaware 

94 

2215 

93 

'24 

85 

2099 

'.972 

12 

1,984 

District  of  Columbia 

95 

2949 

71 

143 

155 

2.722 

2539 

35 

2574 

DoDEA/DDESS 

95 

1 438 

84 

111 

27 

1.384 

1.307 

6 

1,313 

DoDEA/DoDDS 

94 

3.0*0 

172 

354 

73 

2.755 

2583 

21 

2604 

Ficnda 

94 

2091 

H8 

132 

168 

2709 

2543 

6 

2549 

Georgia 

95 

2336 

96 

146 

107 

2 679 

2 530 

12 

2542 

Guam 

94 

1 676 

15 

67 

104 

1 520 

1.415 

16 

1.431 

Hawaii 

95 

2 925 

72 

214 

82 

2.701 

2569 

9 

2578 

■ncfiana 

96 

2 665 

68 

92 

75 

2566 

2469 

1 

2470 

:Owa 

97 

2517 

34 

47 

67 

2437 

2.357 

2 

2359 

Kentucky 

95 

2 349 

67 

125 

89 

2 702 

2561 

18 

2579 

Louisiana 

95 

2964 

71 

’12 

118 

2 805 

2.666 

c. 

2671 

Mere 

94 

2.365 

36 

44 

104 

2 253 

2.109 

6 

2115 

Maryland 

96 

2761 

60 

'39 

107 

2575 

2452 

13 

2465 

Massachusetts 

95 

2 753 

44 

56 

128 

2611 

2488 

9 

2497 

MiCk--QO'' 

94 

2 653 

59 

6C 

88 

2544 

2364 

18 

2382 

Minnesota 

94 

2686 

31 

73 

85 

2559 

2 392 

33 

2425 

Vr.SSSJQp- 

V5 

2927 

9' 

76 

2836 

2 705 

11 

2716 

M'SSOun 

95 

2857 

77 

■’4 

83 

2 777 

2595 

48 

2643 

Montano 

96 

2 41 5 

35 

5? 

56 

2 340 

2 235 

'6 

2251 

Nebraska 

95 

2 923 

49 

7.3 

78 

2 82! 

2 676 

2 

2678 

Nevada 

92 

2 539 

39 

!£■■ 

‘■20 

2.377 

2 177 

16 

2193 

New  Jersev 

95 

2154 

38 

58 

66 

2.068 

1 943 

18 

1.961 

New  Mecco 

94 

277  7 

6V 

' ’ t 

191 

; M2 

2.3’9 

'0 

2389 

New  York 

94 

2.517 

2 1 

37 

• Pi 

/ 386 

2 234 

14 

2248 

North  Carolina 

96 

2922 

62 

9' 

114 

2 779 

2.629 

29 

2658 

Nc^  Dakota 

96 

2 842 

30 

42 

53 

2 772 

2.643 

23 

2666 

C'egon 

95 

2495 

V 

105 

'21 

2 349 

2.223 

10 

2233 

PennsyVania 

95 

256-3 

38 

67 

2469 

2315 

32 

2347 

\ncde  is'ona 

95 

2 746 

46 

9? 

98 

2601 

2 461 

0 

2 461 

South  Carciiro 

95 

2 584 

' J 

96 

7 7 

2.432 

2349 

15 

2364 

Tennessee 

96 

2 716 

55' 

9fl 

96 

2 577 

2 440 

33 

2473 

Texas 

96 

2740 

90 

’25 

’76 

2 527 

2364 

29 

2413 

utan 

95 

2660 

75 

"6 

86 

2?5» 

2566 

59 

2 625 

Vermon* 

96 

2 318 

49 

47 

p: 

2 240 

2*12 

24 

2136 

Vi'gh‘0 

95 

2 S66 

35 

'31 

02 

2 7'8 

2.583 

3 

2586 

Washington 

94 

2883 

73 

’2 

81 

2 808 

2 630 

»0 

2640 

West  Wg.n-a 

95 

2.7e4 

V 

95 

'18 

2 647 

2 522 

8 

2530 

Wsconsin 

95 

2685 

31 

43 

5f>9 

7564 

2 424 

'3 

2437 

Wyoming 

96 

2.963 

73 

94 

67 

2.076 

2 708 

50 

2758 

See  preceding  text  for  explanations  of  the  guidelines  about  sample  representativeness  and  for  the  derivation  of 
weighted  participation.  For  the  national  sample.  Column  3 is  not  applicable. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  19% 
Mathematics  Assessment. 
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Table  B-4 

Student  Participation  Rates,  Grade  4,  1996  Mathematics  Assessment,  Nonpublic  Schools 


Weighted 

P»rc#ntag* 


Nonpublic 

Schools 

Shxtent 

PafUdpaflon 

After 

Make-Ups 

Number  of 
Skxtenft  In 
Original 
Sompte 

Nunb*r  of 
Sluctenhi- 
Sapptem#n:af 
Sampte 

Nunb#r  of 
Student* 
Withdrawn 

NATION 

97 

1.470 

8 

JURISDICTIONS 

Alabama 

97 

248 

1 

2 

Arizona 

99 

196 

0 

5 

Arkansas 

97 

184 

2 

4 

California 

90 

263 

3 

3 

Colorado 

96 

186 

2 

1 

Connecticut 

96 

258 

1 

0 

Delaware 

95 

356 

3 

3 

District  of  Columbia 

96 

413 

1 

2 

Florida 

96 

245 

5 

6 

Georgia 

94 

260 

22 

14 

Guam 

94 

345 

1 

6 

Indiana 

96 

312 

2 

4 

Iowa 

96 

297 

0 

0 

Kentucky 

97 

316 

5 

10 

Louisiana 

97 

460 

8 

3 

Maine 

97 

105 

0 

0 

Maryland 

98 

276 

1 

2 

Massachusetts 

96 

322 

0 

1 

Michigan 

97 

353 

3 

3 

Minnesota 

96 

291 

0 

0 

Mississippi 

96 

276 

7 

3 

Missouri 

95 

472 

3 

5 

Montana 

95 

178 

4 

0 

Nebraska 

99 

443 

3 

5 

Nevada 

96 

161 

1 

2 

New  JerseN 

94 

349 

4 

0 

New  Mexi'. 

94 

225 

18 

13 

New  York 

96 

523 

2 

6 

North  Dakota 

95 

158 

4 

1 

3 

Oregon 

96 

74 

1 

Pennsylvania 

96 

423 

1 

4 

Texas 

96 

107 

2 

1 

Utah 

95 

154 

2 

3 

Vermont 

97 

148 

2 

0 

Wisconsin 

97 

496 

2 

4 

Wyoming 

96 

91 

1 

3 

Numbw  of 


Mumb*rof 

Number  of 
Stuctenh 

Stuctenh 

AtMtMd 

Number  of 
Stuctenh 

Total  Number 

Stuctent* 

TO  So 

Irtflal 

ASMIMd 

of  Stuctenh 

ExchKted 

Amss*d 

Sessions 

Mate  •-Ups 

AsMited 

7 

1.455 

1.394 

18 

1.412 

0 

247 

239 

0 

239 

3 

188 

185 

0 

185 

2 

180 

174 

0 

174 

0 

263 

256 

0 

256 

4 

183 

164 

10 

174 

2 

257 

245 

0 

745 

2 

354 

334 

3 

337 

0 

412 

391 

4 

395 

2 

242 

230 

2 

232 

1 

267 

247 

4 

251 

4 

336 

317 

0 

317 

1 

309 

297 

0 

297 

1 

296 

269 

15 

284 

2 

309 

300 

0 

300 

5 

460 

444 

0 

444 

0 

105 

101 

0 

101 

0 

275 

269 

0 

269 

2 

319 

305 

0 

305 

0 

353 

342 

0 

342 

1 

290 

277 

0 

277 

0 

280 

268 

0 

268 

1 

469 

444 

5 

449 

0 

182 

173 

0 

173 

l 

440 

433 

0 

433 

0 

180 

173 

0 

173 

0 

353 

334 

0 

334 

7 

223 

212 

0 

212 

1 

518 

495 

0 

495 

1 

160 

152 

0 

152 

0 

72 

60 

0 

69 

3 

417 

401 

0 

401 

2 

106 

101 

0 

101 

0 

153 

146 

0 

146 

0 

150 

145 

0 

145 

0 

494 

480 

0 

480 

1 

88 

84 

0 

84 

See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the 
derivation  of  weighted  participation.  For  the  national  sample.  Column  3 is  not  applicable. 


SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-5 

Summary  of  School  and  Student  Participation,  Grade  4 
1996  Mathematics  Assessment,  Public  Schools 


Public 

School* 

Weighted 

Percentage 

School 

Participation 

Befor# 

SubatHution 

Notation 
Number  1 

Walghtad 

Percentage 

School 

Participation 

After 

SubatHution 

Notation 
Number  4 

Notation 
Number  6 

Weighted 

Percentage 

Student 

Participation 

After 

Make-Up* 

Notation 
Number  8 

Weighted 
Overall  Rate 

NATION 

83 

83 

95 

79 

JURISDICTION 

Alabama 

79 

93 

96 

90 

Alaska 

91 

91 

* 

91 

83 

Arizona 

87 

87 

95 

82 

Arkcnsas 

76 

78 

• 

96 

74 

California 

80 

94 

94 

89 

Colorado 

99 

99 

95 

94 

Connecticut 

TOO 

100 

96 

96 

Delaware 

100 

100 

94 

94 

District  of  Columbia 

100 

100 

96 

95 

DoDEA/DOESS 

100 

100 

95 

95 

DoDEA/DoDDS 

100 

100 

94 

94 

Florida 

100 

100 

94 

94 

Georgia 

98 

98 

95 

93 

Guam 

100 

100 

94 

94 

Hawaii 

100 

100 

95 

95 

Indiana 

87 

91 

96 

87 

iowo 

79 

87 

• 

97 

84 

Kentucky 

88 

96 

95 

92 

Louisiana 

100 

100 

95 

95 

Maine 

87 

67 

94 

81 

Maryland 

93 

93 

96 

89 

Massachusots 

97 

97 

95 

92 

Michigan 

76 

88 

• 

94 

82 

Minnesota 

91 

93 

94 

88 

Miss.$s«ppi 

92 

97 

96 

93 

Missouri 

96 

99 

95 

95 

Montana 

70 

81 

• 

96 

79 

Nebraska 

100 

IX 

95 

95 

Nevoda 

84 

86 

* 

92 

79 

Now  Jersey 

73 

73 

• 

95 

69 

New  Mexico 

100 

IX 

94 

94 

New  Yorit 

73 

86 

• 

94 

00 

North  Carolina 

97 

97 

96 

93 

North  Dakota 

75 

96 

96 

92 

Oregon 

86 

90 

95 

85 

Pennsylvania 

73 

86 

• 

95 

01 

Rhode  island 

89 

99 

95 

94 

South  Carolina 

87 

88 

• 

95 

84 

Tennessee 

94 

94 

96 

90 

Texas 

95 

97 

96 

93 

Utah 

100 

IX 

95 

95 

Vermont 

78 

81 

* 

96 

77 

Virginia 

100 

IX 

95 

95 

Washington 

99 

99 

94 

93 

West  Virginia 

100 

IX 

96 

95 

Wisconsin 

92 

94 

95 

90 

Wyoming 

100 

IX 

96 

96 

See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the  derivation  of  weighted 
participation. 

Notation  Number  1 : The  state’s  public  school  weighted  participation  rate  for  the  initial  sample  was  less  than  70%. 

Notation  Number  4:  The  state’s  public  school  weighted  participation  rate  for  the  initial  sample  of  schools  was  below  85%  and  the 
weighted  school  participation  rate  after  substitution  was  below  90%. 


Notation  Number  6:  The  nonparticipating  public  schools  included  a class  of  schools  with  similar  characteristics,  which  together 
accounted  for  more  than  5%  of  the  state’s  total  fourth-grade  weighted  sample  of  public  schools. 

Notation  Number  8:  The  weighted  student  response  rate  within  participating  public  schools  was  below  85%. 

SOURCE  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP).  1996  Mathematics 
Assessment 


260 


o o o 


BEST  COPY  AVAILABLE 


Table  B-6 

Summary  of  School  and  Student  Participation,  Grade  4 
1996  Mathematics  Assessment,  Nonpublic  Schools 


Weighted  Weighted  Weighted 

Percentage  Percentage  Percentage 

School  School  Student 

Participation  Participation  Participation 

Nonpublic  Before  Notation  After  Notation  Notation  Ater  Notation  Weighted 

Schools Substitution  Number 2 Substitution  Numbers  Number 7 Mtke-Upa  Number 9 Overall  Rate 

NATION  79  79  97  77 


JURISDICTIONS 


Alabama 

72 

72 

Arizona 

78 

87 

Arkansas 

86 

86 

California 

73 

73 

Colorado 

76 

76 

Connecticut 

75 

75 

Delaware 

41 

* 

43 

District  of  Columbia 

63 

* 

66 

Florida 

66 

* 

73 

Geoigia 

99 

99 

Guam 

78 

78 

Indiana 

79 

86 

Iowa 

82 

82 

Kentucky 

87 

87 

Louisiana 

86 

86 

Maine 

71 

74 

Maryland 

57 

• 

57 

Massachusetts 

84 

84 

Michigan 

86 

94 

Minnesota 

76 

78 

Mississippi 

79 

79 

Missouri 

99 

100 

Montana 

94 

94 

Nebraska 

91 

91 

Nevada 

91 

100 

New  Jersey 

64 

* 

75 

New  Mexico 

90 

90 

New  York 

83 

91 

North  Dakota 

68 

■ 

68 

Oregon 

34 

* 

34 

Pennsylvania 

66 

* 

66 

Texas 

64 

* 

64 

Utah 

81 

81 

Vermont 

74 

74 

Wisconsin 

68 

* 

73 

Wyoming 

82 

95 

97 

99 

97 

98 
96 
96 

95 

96 
96 
94 

94 
96 

96 

97 
97 

97 

98 

96 

97 
96 
96 

95 

95 

99 

96 
94 

94 
96 

95 

96 
96 

96 

95 

97 
97 

96 


69 

86 

83 

72 

73 

71 
41 
63 
69 
92 

74 

83 
79 

84 
83 

72 
56 
81 
91 

75 

76 

95 

89 

90 

96 

70 

85 
87 
64 
33 
63 
61 

77 
72 

71 
92 


See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the 
derivation  of  weighted  participation. 

Notation  Number  2:  The  state’s  nonpublic  school  weighted  participation  rate  for  the  initial  sample  was  less  than  70%. 


Notation  Number  5:  The  state’s  nonpublic  school  weighted  participation  rate  for  the  initial  sample  ot  schools  was 
beiow  85%  and  the  weighted  school  participation  rate  after  substitution  was  below  90%. 

Notation  Number  7:  The  nonparticipating  nonpublic  schools  included  a class  of  schools  with  similar  characteristics, 
which  together  accounted  for  more  than  5%  of  the  state’s  total  fourth-grade  weighted  sample  of  nonpublic  schools. 

Notation  Number  9:  The  weighted  student  response  rate  within  participating  nonpublic  schools  was  below  85%. 

SOURCE:  National  Center  for  Education  Statistics.  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-7 

Weighted  Percentages  of  Students  Excluded  (SD  and  LEP)from  Student  Sample,  Grade  4 
1996  Mathematics  Assessment,  Public  Schools 


Public 

School* 

Total 

Percentage 
Students 
Identified  SD 
and  LEP 

Total 

Percantage 

Students 

Excluded 

Percentage 

Students 

Identified 

SD 

Percentage 

Students 

Excluded 

SD 

Percentage 

Students 

Identified 

LEP 

Percentage 

Student* 

Excluded 

LEP 

NATION 

15 

6 

11 

5 

4 

2 

JURISDICTIONS 

Alabama 

U 

6 

11 

6 

0 

0 

Alaska 

21 

4 

13 

3 

9 

1 

Ari2ona 

22 

13 

10 

7 

13 

7 

Arkansas 

9 

7 

9 

6 

0 

0 

California 

33 

16 

9 

6 

26 

13 

Colorado 

16 

9 

13 

7 

4 

2 

Connecticut 

15 

6 

12 

6 

3 

2 

Delaware 

14 

7 

11 

5 

2 

2 

District  of  Columbia 

14 

11 

9 

7 

6 

5 

DoDE  A/DDE  SS 

9 

4 

8 

3 

1 

1 

DoDEA/DoDDS 

10 

5 

9 

4 

2 

1 

Florida 

19 

10 

14 

7 

6 

3 

Georgia 

13 

7 

12 

6 

2 

1 

Guam 

16 

13 

7 

6 

9 

7 

Hawaii 

14 

6 

9 

4 

5 

1 

indtona 

11 

5 

11 

5 

1 

0 

Iowa 

12 

5 

10 

4 

2 

1 

Kentucky 

10 

6 

10 

6 

0 

0 

Louisiana 

13 

7 

13 

7 

1 

0 

Maine 

16 

8 

15 

8 

0 

0 

Maryland 

14 

8 

13 

7 

1 

1 

Massac  husets 

16 

8 

14 

7 

2 

2 

Michigan 

12 

7 

10 

6 

2 

1 

Minnesota 

13 

6 

10 

4 

3 

1 

Mississippi 

7 

5 

7 

5 

0 

0 

Missouri 

15 

5 

14 

5 

1 

0 

Montana 

10 

5 

10 

5 

0 

0 

Nebraska 

16 

5 

14 

5 

2 

1 

Nevodo 

15 

8 

10 

6 

6 

3 

New  Jersey 

10 

6 

9 

5 

2 

1 

New  Mexico 

22 

12 

14 

8 

10 

5 

New  York 

17 

9 

10 

6 

7 

4 

North  Carolina 

15 

7 

13 

7 

2 

1 

North  Dakota 

11 

4 

10 

3 

0 

0 

Oregon 

20 

9 

13 

6 

7 

3 

Pennsylvania 

10 

5 

9 

4 

1 

1 

RixDde  Island 

53 

6 

13 

5 

5 

2 

South  Carolina 

13 

6 

12 

6 

0 

0 

Tennessee 

14 

7 

12 

6 

1 

1 

Tsxos 

25 

11 

13 

8 

14 

5 

Utah 

13 

6 

11 

5 

2 

Vermont 

14 

6 

14 

6 

1 

0 

Virginia 

14 

7 

12 

6 

2 

! 

Washington 

14 

6 

11 

5 

3 

1 

West  Virginia 

13 

8 

13 

8 

0 

0 

Wisconsin 

12 

8 

11 

8 

2 

1 

Wyoming 

12 

4 

12 

4 

1 

0 

SD  = Students  with  Disabilities  (the  term  previously  used  was  IEP). 

LEP  = Limited  English  Proficiency. 

To  be  excluded,  a student  was  supposed  to  be  classified  as  SD  or  as  LEP  and  judged  incapable  of  participating  in  the 
assessment.  A student  reported  as  belonging  to  both  SD  and  LEP  classifications  is  counted  once  in  the  overall  rate  (first 
column),  once  in  the  overall  excluded  rate  (second  column),  and  separately  in  the  remaining  columns. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  19% 
Mathematics  Assessment. 
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Table  B-8 

Weighted  Percentages  of  Students  Excluded  (SD  and  LEP)  from  Student  Sample,  Grade  4 
1996  Mathematics  Assessment,  Nonpublic  Schools 


Nonpubtic 

Schools 

Total 

Percentage 
Students 
Identified  SD 
and  LEP 

Total 

Percentage 

Students 

Excluded 

Percentage 

Students 

Identified 

SD 

Percentage 

Students 

Excluded 

SD 

Percentage 

Students 

Identified 

LEP 

Percentage 

Students 

Excluded 

LEP 

NATION 

2 

l 

2 

l 

0 

0 

JURISDICTIONS 

Alobama 

0 

0 

0 

0 

0 

0 

Arizona 

43 

3 

3 

3 

43 

3 

Arkansas 

4 

2 

2 

1 

1 

1 

California 

0 

0 

0 

0 

0 

0 

Colorado 

5 

5 

5 

5 

0 

G 

Connecticut 

7 

1 

5 

0 

3 

1 

Delaware 

l 

1 

1 

1 

0 

0 

District  of  Columbia 

t 

0 

0 

0 

1 

0 

Florida 

1 

1 

1 

I 

0 

0 

Georgia 

5 

2 

5 

2 

0 

0 

Guam 

3 

2 

0 

0 

3 

2 

indfana 

3 

1 

2 

0 

1 

1 

Iowa 

2 

1 

2 

1 

0 

0 

Kentucky 

2 

1 

2 

1 

0 

0 

Louisiana 

7 

2 

7 

1 

0 

0 

Maine 

3 

0 

3 

0 

0 

0 

Maryland 

0 

0 

0 

0 

0 

0 

Massachusetts 

6 

1 

6 

l 

0 

0 

Michigan 

3 

0 

2 

0 

0 

0 

Minnesota 

5 

1 

5 

1 

0 

0 

Mississippi 

8 

c 

8 

0 

0 

0 

Missouri 

3 

0 

3 

0 

0 

0 

Montana 

0 

0 

0 

0 

0 

0 

Nebraska 

2 

0 

1 

0 

1 

0 

Nevada 

0 

0 

0 

0 

0 

0 

New  Jersey 

3 

0 

3 

0 

0 

0 

New  Mexico 

15 

7 

7 

7 

12 

4 

New  York 

14 

1 

1 

1 

14 

1 

North  Dakota 

4 

2 

4 

2 

0 

0 

Oregon 

0 

0 

0 

0 

0 

0 

Pennsylvania 

2 

1 

2 

1 

0 

0 

Texas 

3 

3 

3 

3 

0 

0 

Utah 

0 

0 

0 

0 

0 

0 

Vermont 

0 

0 

0 

0 

0 

0 

Wisconsin 

0 

0 

0 

0 

0 

0 

Wyoming 

4 

4 

4 

4 

0 

0 

SD  =s  Students  with  Disabilities  (the  term  previously  used  was  IEP). 

LEP  = Limited  English  Proficiency. 

To  be  excluded,  a student  was  supposed  to  be  classified  as  SD  or  as  LEP  and  judged  incapable  of  participating  in  the 
assessment.  A student  reported  as  belonging  to  both  SD  and  LEP  classifications  is  counted  once  in  the  overall  rate  (first 
column),  once  in  the  overall  excluded  rate  (second  column),  and  separately  in  the  remaining  columns. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-9 

Weighted  Percentages  of  Absent,  SD,  and  LEP  Students  Based  on  Those  Invited  to  Participate 
Grade  4,  1996  Mathematics  Assessment,  Public  Schools 


Public 

School# 

Weighted 

Percentage 

Student 

Participation 

After 

Make-Ups 

Weighted 

Percentage 

Absent 

Weighted 

Percentage 

Assessed 

SD 

Weighted 

Percentage 

Absent 

SD 

Weighted 

Percentage 

Assessed 

LEP 

Weighted 

Percentage 

Absent 

LEP 

NATION 

95 

5 

94 

6 

96 

4 

JURISDICTIONS 

AJaooma 

96 

4 

93 

7 

100 

0 

Moska 

91 

9 

93 

7 

92 

8 

Af'ZOfYC 

95 

5 

94 

6 

96 

4 

Arkansas 

96 

4 

98 

2 

0 

100 

Californio 

94 

6 

92 

8 

97 

3 

Co'cvaao 

95 

5 

93 

7 

92 

8 

Connecticut 

96 

4 

95 

5 

95 

5 

De'awore 

94 

6 

93 

7 

88 

12 

D-sf'ict  of  Co!urnt>a 

95 

5 

95 

5 

97 

3 

DoDcA/DDESS 

95 

t 

94 

6 

100 

0 

DoCFA/CcDDS 

94 

6 

95 

5 

100 

0 

-icr'dO 

94 

a 

90 

10 

93 

7 

George 

95 

6 

90 

10 

91 

9 

Gu3ni 

94 

6 

100 

0 

93 

7 

Hawai 

95 

5 

92 

8 

94 

6 

m.-j  ano 

96 

4 

94 

6 

0 

100 

ewa 

97 

y 

96 

4 

100 

0 

95 

5 

98 

2 

0 

100 

lO^SOna 

95 

r„ 

95 

5 

100 

0 

V.O  ne 

94 

6 

91 

9 

100 

0 

MarvOnO 

96 

4 

96 

4 

100 

0 

Vas$achuse»s 

96 

5 

91 

9 

100 

0 

V c^gan 

94 

6 

9C 

10 

8> 

19 

V-nnesa’a 

94 

6 

87 

13 

97 

3 

V SS-SSlCC- 

96 

4 

97 

3 

0 

100 

M SSCuf! 

95 

5 

97 

3 

100 

0 

Von*  ana 

96 

4 

97 

3 

100 

0 

Neb'oska 

95 

5 

93 

7 

83 

17 

Nevada 

9? 

£ 

92 

8 

87 

13 

New  .©'sev 

96 

£ 

92 

8 

100 

0 

New  Vexicc 

94 

A 

96 

4 

98 

2 

Vo'K 

94 

6 

66 

M 

91 

9 

N.'/*n  Co’r-  na 

96 

4 

97 

3 

100 

0 

Nofin  Oa<a‘:’) 

96 

4 

94 

6 

100 

0 

Oregon 

95 

5 

95 

5 

95 

5 

Ponrsyivan-a 

95 

£ 

*s 

93 

48 

52 

Yrccs  isana 

95 

6 

97 

3 

94 

6 

Sa./*i  CO'O'-na 

95 

£ 

96 

4 

■00 

0 

•e-nessee 

96 

4 

91 

9 

80 

20 

'■?»  n 

?6 

4 

98 

2 

97 

3 

/ a*- 

95 

£ 

90 

10 

100 

0 

o'Tvy't 

96 

4 

v-* 

7 

100 

0 

V 'g  n a 

95 

f. 

93 

; 

100 

0 

v\asn.'  'g»or 

94 

* 

57 

13 

98 

2 

vVes*  V.*gn.o 

95 

6 

69 

IV 

100 

0 

Ws:.^ns  r* 

VC 

5 

94 

6 

96 

4 

Wyr  'nog 

96 

4 

93 

7 

100 

0 

SD  = Students  with  Disabilities  (the  term  previously  used  was  IEP). 

LEP  = Limited  English  Proficiency. 

SOl'RCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-10 

Weighted  Percentages  of  Absent,  SD , and  LEP  Students  Based  on  Those  Invited  to  Participate 
Grade  4,  1996  Mathematics  Assessment,  Nonpublic  Schools 


Nonpublic 

Schools 

Weighted 

Percentage 

Student 

Participation 

After 

Make-Ups 

Weighted 

Percentage 

Absent 

Weighted 

Percentage 

Assessed 

SD 

Weighted 
Percentage 
Absent  SD 

Weighted 

Percentage 

Assessed 

LEP 

Weighted 
Percentage 
Absent  LEP 

NATION 

97 

3 

86 

14 

- 

- 

JURISDICTIONS 

Alabama 

97 

3 

0 

too 

0 

too 

Arizona 

99 

1 

0 

too 

too 

0 

Arkansas 

97 

3 

too 

0 

0 

too 

California 

98 

2 

0 

too 

0 

too 

Colorado 

96 

4 

0 

too 

0 

too 

Connecticut 

96 

4 

too 

0 

too 

0 

Delaware 

95 

5 

0 

too 

0 

too 

District  of  Columbia 

96 

4 

0 

too 

too 

0 

Florida 

96 

4 

0 

too 

0 

too 

Georgia 

94 

6 

too 

0 

0 

too 

Guam 

94 

6 

0 

too 

too 

0 

Indiana 

96 

4 

too 

0 

0 

too 

Iowa 

96 

4 

too 

0 

0 

too 

Kentucky 

97 

3 

50 

50 

0 

too 

Louisiana 

97 

3 

92 

8 

0 

too 

Maine 

97 

3 

too 

0 

0 

too 

Maryland 

98 

2 

0 

too 

0 

too 

Massachusetts 

96 

4 

too 

0 

0 

too 

Michigan 

97 

3 

100 

0 

too 

0 

Minnesota 

96 

4 

too 

0 

0 

too 

Mississippi 

96 

4 

too 

0 

0 

too 

Missouri 

95 

5 

84 

16 

0 

too 

Montana 

95 

5 

0 

too 

0 

too 

Nebraska 

99 

1 

too 

0 

too 

0 

Nevada 

96 

4 

0 

too 

0 

too 

New  Jersey 

94 

6 

67 

33 

0 

too 

New  Mexico 

94 

6 

0 

too 

88 

13 

New  York 

96 

4 

0 

too 

too 

0 

North  Dakota 

95 

5 

too 

0 

0 

too 

Oregon 

96 

4 

0 

too 

0 

too 

Pennsylvania 

96 

4 

too 

0 

0 

100 

Texas 

96 

4 

0 

too 

0 

too 

Utah 

95 

5 

0 

too 

0 

too 

Vermont 

97 

3 

0 

too 

0 

too 

Wisconsin 

97 

3 

0 

too 

0 

too 

Wyoming 

96 

4 

0 

too 

0 

too 

SD  = Students  with  Disabilities  (the  term  previously  used  was  IEP). 

LEP  = Limited  English  Proficiency. 

The  dash  (-)  indicates  that  there  were  no  students  in  that  category. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  199b 
Mathematics  Assessment. 
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Table  B-ll 

Questionnaire  Response  Rates ; Grade  4 , 7996  Mathematics  Assessment ; Public  Schools 


Public 

Schools 

Weighted  Percentage 
of  Student*  Matched 
to  Mathematic*  Teecher 
Questionnaire* 

Weighted  Percentage 
of  Students  Matched 
to  School 
Characteristic*/ 
Policies  Queetionnarie 

Percentage  of  School 
Characteristic*/ 
Policies 
Questionnaires 
Returned 

Percentage  of 
SCVLEP  Student' 
Questionnaires 
Returned 

NATION 

96 

96 

96 

85 

JURISDICTIONS 

Alabama 

97 

92 

93 

95 

Alaska 

80 

67 

88 

63 

Aicano 

89 

95 

95 

87 

Arkansas 

97 

88 

89 

93 

Californio 

93 

84 

86 

83 

C oorodo 

97 

93 

93 

93 

C onnocltcui 

98 

91 

90 

95 

Delaware* 

95 

98 

96 

95 

Dis»nci  of  CoJumbio 

93 

88 

89 

87 

DoDt'A/DOESS 

95 

87 

84 

94 

DoOtA/DtODS 

94 

97 

96 

92 

r lendo 

96 

B6 

87 

89 

Georgia 

98 

92 

92 

92 

Guam 

89 

90 

95 

77 

Hawa« 

95 

89 

89 

93 

I'KJiona 

<77 

92 

93 

90 

95 

93 

93 

93 

k Unlucky 

94 

91 

92 

94 

i '‘•u'S'Oria 

99 

88 

90 

85 

Womtr 

95 

96 

95 

90 

Ma»  /and 

97 

89 

90 

90 

Massarhusois 

97 

86 

85 

90 

M'chgan 

97 

88 

60 

93 

M.nnnsola 

96 

87 

89 

69 

Mississippi 

96 

92 

92 

92 

M-ssou'i 

91 

68 

87 

94 

Montana 

93 

92 

93 

91 

Nebraska 

97 

92 

92 

94 

Nevada 

97 

90 

92 

91 

Now  JersQ-. 

98 

94 

04 

93 

New  Mexii*  a 

92 

90 

90 

85 

No  .v  v,vk 

97 

96 

96 

90 

N (.  a*  ai.no 

96 

95 

95 

91 

Nr  r*K  Dakota 

95 

96 

95 

90 

' 

95 

92 

94 

ee 

I'ponsyKoma 

97 

90 

90 

90 

f^t'oejo  island 

99 

92 

91 

94 

outn  L.d'Oino 

9* 

89 

89 

91 

r.mnnsspp 

97 

94 

94 

89 

Texas 

92 

89 

88 

8/ 

l-l  jh 

94 

92 

92 

90 

vo**nant 

98 

92 

91 

90 

Vyg.n.U 

97 

95 

96 

97 

VVnsb-r\)ton 

97 

93 

94 

92 

vVost  Vngiri<a 

95 

90 

89 

93 

Wisconsin 

94 

88 

88 

68 

AS  M^ung 

95 

95 

92 

91 

1 This  percentage  is  unweighted,  and  computed  over  all  students,  whereas  all  other  numbers  in  this  report  arc  based  on 
(he  reporting  sample  only. 


SOl'RCE:  National  Center  for  Education  Statistics.  National  Assessment  of  Educational  Progress  (NAEP).  1996 
Mathematics  Assessment. 
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Table  B-12 

Questionnaire  Response  Rates,  Grade  4,  1996  Mathematics  Assessment,  Nonpublic  Schools 


Nonpubiic 

Schools 

Weighted  Percentage 
of  Students  Matched 
to  Mathematics  Teacher 
Questionnaires 

Weighted  Percentage 
of  Students  Matched 
to  School 
Characteristics/ 
Policies  Questionnaire 

Percentage  of  School 
Characteristics/ 
Potlcies 
Questionnaires 
Returned 

Percentage  of 
SD/LEP  Student’ 
Questionnaires 
Returned 

NATION 

89 

95 

94 

90 

JURISDICTIONS 

Alabama 

100 

100 

100 

0 

Arizona 

92 

98 

90 

13 

Arkansas 

100 

100 

100 

75 

Californio 

98 

100 

100 

0 

Colorado 

94 

89 

90 

89 

Connecticut 

100 

82 

85 

95 

Delaware 

90 

85 

85 

100 

District  of  Columbia 

83 

74 

72 

100 

Florida 

96 

88 

92 

100 

•Georgia 

97 

92 

92 

6 

Guam 

84 

90 

89 

80 

Indiana 

91 

91 

93 

80 

Iowa 

88 

84 

80 

89 

Kentucky 

90 

71 

77 

0 

Louisiana 

100 

94 

95 

100 

Marne 

100 

82 

88 

100 

Maryland 

100 

76 

82 

100 

Massachusetts 

90 

100 

100 

100 

Michigan 

97 

81 

78 

89 

Minnesota 

92 

100 

100 

75 

Mississippi 

89 

88 

91 

100 

Missouri 

99 

90 

91 

100 

Montana 

100 

100 

100 

83 

Nebraska 

86 

91 

91 

78 

Nevada 

100 

100 

100 

0 

New  Jersey 

89 

96 

94 

95 

New  Mexico 

68 

87 

85 

48 

New  York 

91 

79 

62 

24 

North  Dakota 

84 

77 

82 

97 

Oregon 

75 

100 

100 

0 

Pennsylvania 

94 

96 

95 

IX 

Texos 

100 

87 

75 

0 

Utah 

100 

95 

86 

0 

Vermont 

98 

66 

67 

80 

Wisconsin 

100 

93 

92 

0 

Wyoming 

100 

100 

100 

67 

'This  percentage  is  unweighted,  and  computed  over  all  students,  whereas  all  other  numbers  in  this  report  arc  based  on 
the  reporting  sample  only. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-13 

Participation  Rates,  Grade  8,  1996  Mathematics  Assessment,  Public  Schools 


Weighted 

Percentage* 

School 

Weighted 

Percentage 

School 

Participate** 

After 

SubeMutton 

Number  ot 
School*  in 
Original 
Sample 

Number  ot 
School*  in 
Oritfnat 
Sample  that 
Participated 

Numbtrot 

Number  ot 
Substituted 
School*  that 
Pcrtctpoled 

Totctf 

Number  ot 
School*  that 

n „Mi  In  rJ*  1 

raroapcMa 

Pufc*c 

School* 

More 

SubeWuHon 

School*  not 
Hgfcte 

School* 

Provided 

NATION 

81 

02 

228 

0 

191 

5 

1 

192 

JURISDICTIONS 

Alabama 

84 

90 

110 

2 

91 

15 

6 

97 

Alaska’ ' 

92 

92 

80 

8 

53 

17 

0 

53 

Arizona 

87 

87 

107 

1 

93 

9 

0 

93 

Arkansas' 

70 

71 

112 

1 

76 

25 

1 

77 

Cakfomia 

83 

94 

108 

0 

89 

19 

12 

101 

Colorado 

100 

100 

109 

1 

108 

0 

0 

108 

Connecticut 

100 

100 

103 

1 

102 

0 

0 

102 

Delaware 

100 

100 

31 

1 

30 

0 

0 

30 

Dslnct  of  Colombo 

100 

100 

35 

3 

32 

0 

0 

32 

DoOEA/DOESS 

100 

100 

12 

0 

12 

0 

0 

12 

DcOEA/DoDDS 

100 

100 

59 

2 

57 

0 

0 

57 

Florida 

100 

100 

109 

5 

104 

0 

0 

104 

Georg  o 

99 

99 

108 

7 

100 

1 

0 

100 

Guam 

100 

100 

6 

0 

6 

0 

0 

6 

HOwOi 

100 

100 

53 

1 

51 

0 

0 

51 

indcano 

88 

91 

107 

1 

93 

12 

3 

96 

l0wO4 

74 

84 

i 14 

3 

82 

27 

n 

93 

Kentucky 

68 

92 

no 

1 

96 

9 

5 

101 

toms  >ana 

100 

100 

114 

2 

112 

0 

0 

112 

Maine 

90 

90 

no 

6 

93 

6 

0 

93 

Moniond* 

86 

86 

106 

2 

89 

10 

0 

89 

Massachusetts 

92 

92 

106 

3 

98 

7 

0 

98 

Michigan’ 

70 

86 

106 

0 

74 

31 

16 

90 

Minnesota 

86 

88 

108 

0 

94 

8 

2 

96 

Mississippi 

89 

95 

109 

3 

96 

9 

7 

103 

Missouri 

93 

96 

117 

7 

102 

7 

3 

105 

Montana' 

72 

75 

112 

10 

69 

22 

6 

75 

Nebraska 

99 

100 

132 

14 

115 

3 

1 

116 

Nevada 

38 

38 

59 

2 

28 

2 

0 

28 

New  Hampshire 

66 

69 

87 

0 

59 

11 

3 

62 

New  Jersey 

tA 

65 

109 

2 

68 

34 

1 

69 

New  Mexco 

ICO 

100 

90 

0 

9C 

0 

0 

90 

New  vofi/ 

71 

80 

106 

0 

75 

29 

9 

84 

North  Caroitno 

100 

100 

108 

1 

107 

0 

0 

107 

Nonh  Dakota 

83 

95 

124 

7 

96 

17 

12 

108 

Oregon 

86 

92 

in 

4 

92 

13 

6 

98 

Rhode  tsiand 

90 

90 

51 

1 

42 

4 

0 

42 

South  Cardna 

86 

67 

107 

2 

90 

10 

1 

91 

Tennessee 

92 

92 

112 

5 

98 

6 

0 

98 

Texas 

90 

95 

no 

5 

95 

10 

5 

100 

Utah 

100 

100 

97 

2 

95 

0 

0 

95 

Vermont' 

74 

74 

105 

6 

75 

1 

0 

75 

Virginia 

100 

100 

106 

0 

106 

0 

0 

106 

Washrngton 

94 

95 

no 

1 

102 

7 

1 

103 

West  Vtrgma 

100 

100 

107 

1 

106 

0 

0 

106 

Wisconsin' 

78 

78 

1 14 

0 

90 

23 

0 

90 

Wyoming 

100 

100 

74 

4 

70 

0 

0 

70 

See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the  derivation  of  weighted  participation.  For 
Delaware,  the  District  of  Columbia.  DDESS.  DoDDS.  Guam.  Hawaii,  and  Rhode  Island,  the  State  Assessment  was  based  on  all  eligible  public  schools 
ti  e . there  was  no  sampling  of  public  schools). 

1 The  state’s  public  school  weighted  participation  rate  for  the  initial  sample  was  less  than  70%. 

4 The  state’s  public  school  weighted  participation  rate  for  the  initial  sample  of  schools  was  below  85%  and  the  weighted  school  participation  rate  after 
substitution  was  below  90% 

'‘The  nonparticipating  public  schools  included  a class  of  schools  with  similar  characteristics,  which  together  accounted  for  more  than  5%  of  the  state  s total 
fourth-grade  weighted  sample  of  public  schools. 

"The  weighted  student  response  rate  within  participating  public  schools  was  below  85%. 

In  Alaska.  Massachusetts.  Oregon,  and  Washington,  the  materials  from  one  school  that  conducted  an  assessment  were  lost  in  shipping.  The  school  is 
included  in  the  counts  ot  participating  schools,  both  before  and  after  substitution  However,  in  the  weighted  results,  the  school  is  treated  in  the  same  manner 
as  a nonparticipatinp  school  because  no  student  responses  were  available  for  analysis  and  reporting. 

SOURCE  National  Center  lor  Education  Statistics.  National  Assessment  of  Educational  Progress  tNAE.P).  19%  Mathematics  Assessment 
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Table  B-14 

School  Participation  Rates,  Grade  8,  1996  Mathematics  Assessment,  Nonpublic  Schools 


Weighted 

Weighted 

Percentage 

Percentage 

Number  of 

School 

School 

Number  of 

Schools  in 

Number  of 

Number  of 

Total  Number 

Participation 

Participation 

Schools  In 

Number  of 

Origins! 

Substituted 

Substituted 

of  Schools 

Nonpublic 

Before 

After 

Original 

Schools  not 

Sample  that 

Schools 

Schools  that 

that 

Schools 

Substitution 

Substitution 

Sample 

Eligible 

Participated 

Provided 

Participated 

Participated 

NATION 

81 

81 

105 

9 

78 

0 

0 

78 

JURISDICTIONS 

Alabama’ 

64 

64 

18 

3 

9 

K 

0 

9 

Arkansas’ 

51 

60 

11 

4 

4 

2 

» 

5 

Californio’ 

75 

75 

24 

5 

13 

6 

0 

13 

Connecticut 

63 

65 

35 

7 

18 

9 

1 

19 

Delaware’ 

38 

40 

50 

12 

11 

8 

1 

12 

District  of  Columbia’ 

47 

47 

42 

10 

16 

0 

0 

16 

Georgia 

88 

88 

16 

3 

10 

3 

0 

10 

Guam' 

76 

76 

11 

0 

8 

0 

0 

8 

Iowa 

88 

88 

22 

4 

15 

3 

0 

15 

Kentucky’ 

67 

67 

21 

4 

11 

5 

0 

11 

Louisiana’ 

73 

73 

34 

2 

22 

7 

0 

22 

Maryland' 

60 

64 

34 

5 

17 

12 

1 

18 

Massachusetts’ 

70 

74 

34 

9 

17 

8 

1 

18 

Michigan’ 

80 

88 

27 

6 

16 

5 

2 

18 

Minnesota' 

75 

75 

25 

5 

15 

5 

0 

15 

Missouri 

94 

100 

32 

10 

21 

l 

1 

22 

Montana’ 

78 

78 

19 

7 

9 

3 

0 

9 

Nebraska’ 

83 

85 

31 

5 

20 

5 

0 

20 

Nevada5 

78 

78 

11 

3 

6 

2 

0 

6 

New  Hampshire' 

85 

85 

19 

4 

12 

2 

0 

12 

New  Jersey* 

68 

71 

42 

10 

21 

9 

1 

22 

New  Mexico' 

87 

87 

22 

6 

12 

4 

0 

12 

New  York’ 

88 

90 

39 

6 

29 

3 

1 

30 

North  Dakota 

86 

86 

20 

4 

12 

4 

0 

12 

Oregon’ 

22 

22 

18 

3 

3 

10 

0 

3 

Rhode  Island’ 

81 

81 

39 

6 

26 

3 

0 

26 

South  Ccrdina’ 

76 

76 

16 

2 

10 

4 

0 

10 

Texas 

93 

93 

12 

1 

9 

2 

0 

9 

Utah’ 

43 

43 

in 

3 

2 

4 

0 

2 

Vermont 

73 

73 

n 

9 

2 

0 

9 

Washington 

86 

86 

‘<9 

7 

9 

2 

0 

9 

Wisconsin’ 

68 

73 

50 

8 

26 

14 

2 

28 

Wyoming’ 

74 

74 

12 

5 

5 

2 

0 

5 

See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the 
derivation  of  weighted  participation. 

For  the  District  of  Columbia  and  Guam,  the  State  Assessment  was  based  on  all  eligible  nonpublic  schools  (i.e.,  there 
was  no  sampling  of  nonpublic  schools). 

2 The  state’s  nonpublic  school  weighted  participation  rate  for  the  initial  sample  was  less  than  70%. 

5 The  state’s  nonpublic  school  weighted  participation  rate  for  the  initial  sample  of  schools  was  below  857r  and  the 
weighted  school  participation  rate  after  substitution  was  below  90%. 

7 The  nonparticipating  nonpublic  schools  included  a class  of  schools  with  similar  characteristics,  which  together 
accounted  for  more  than  5%  of  the  stale’s  total  fourth-grade  weighted  sample  of  nonpublic  schools. 

In  Arkansas,  the  District  of  Columbia,  and  Kentucky,  the  materials  from  one  school  that  conducted  an  assessment  were 
lost  in  shipping.  The  school  is  included  in  the  counts  of  participating  schools,  both  before  and  after  substitution. 
However,  in  the  weighted  results,  the  school  is  treated  in  the  same  manner  as  a nonparticipating  school  because  no 
student  responses  were  available  for  analysis  and  reporting. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  19% 
Mathematics  Assessment. 
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Table  B-15 

Student  Participation  Rates.  Grade  8,  1996  Mathematics  Assessment , Public  Schools 


Weighted 

Percentage 

Skident 

Parfclpatton 

After 

Makeup* 


Number  of  Number  of 

Student*  In  Students  In  Number  of 

Orltfnal  Supplemental  Student* 

Sample  Somple  Withdrawn 


JURISDICTIONS 

Alabama 

Alaska 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District  of  Columbia 

DoDEA/DDESS 

Do  DE  A/DoDDS 

Florida 

Georg  10 

Guam 

Howan 

ind-ana 

lowa 

Kentucky 


Louisiana 

8<? 

3.070 

77 

149 

Marne 

92 

2.538 

21 

41 

Maryland 

91 

2.479 

59 

108 

Massachusetts 

92 

2.579 

43 

63 

Michigan 

90 

2 497 

34 

84 

Minnesota 

92 

2.719 

36 

76 

Mississippi 

93 

2.850 

62 

126 

Missouri 

91 

2 789 

58 

122 

Montana 

92 

2 143 

39 

73 

Nebraska 

91 

2 954 

61 

71 

Nevada 

90 

1. 156 

5i 

69 

New  Hampshire 

89 

1 990 

26 

49 

New  Jersey 

93 

1 873 

33 

55 

New  Mexico 

90 

2.854 

74 

174 

New  Vork 

91 

2 275 

20 

52 

North  Carolina 

91 

2.995 

86 

118 

North  Dakoto 

94 

2.824 

28 

50 

Oiegon 

90 

2.684 

81 

127 

Rhode  island 

89 

2.445 

36 

102 

South  Carolina 

89 

2.534 

58 

121 

Tennossee 

91 

2 644 

42 

93 

Texas 

92 

2.636 

83 

174 

Utah 

91 

3 112 

°7 

138 

Vermont 

93 

2.216 

29 

42 

Viig'nia 

91 

2.952 

67 

H3 

Washington 

90 

2.827 

60 

95 

West  Virgmio 

92 

2 972 

64 

113 

Wisconsin 

92 

2 463 

29 

49 

Wyoming 

93 

2.970 

69 

131 

Number  of 
Ski  deni* 


Number  of 
Skident* 

Tofte 


Number  of 

Student*  Number  of 


Aueued 

l ratal 


Student*  Total  Number 
A*sewed  of  Skident* 


eluded 

AUMMd 

SeitJcru 

Make-Up* 

A**e**ed 

162 

6 050 

5 369 

201 

5.590 

106 

2.434 

1209 

52 

2 261 

54 

1.782 

1.452 

10 

1.462 

111 

2.346 

2.086 

50 

1136 

86 

1 995 

1.833 

12 

1.845 

161 

2.505 

1260 

30 

2.290 

62 

2.782 

2.500 

30 

1530 

144 

2.735 

2.417 

68 

1485 

102 

1004 

1.725 

73 

1.798 

102 

1.998 

1.585 

108 

1.693 

16 

650 

620 

0 

620 

27 

2.308 

2.156 

4 

2.160 

144 

2.647 

1360 

41 

2.401 

88 

2.629 

2.315 

49 

2.364 

24 

1.076 

909 

19 

928 

75 

2.409 

2.127 

62 

2.189 

75 

2.528 

2.323 

24 

2.347 

57 

2.315 

2.138 

31 

1169 

75 

2.618 

1391 

70 

1461 

89 

2.909 

1558 

41 

2.699 

53 

2.465 

2.244 

14 

1258 

60 

2.350 

2.104 

33 

1137 

91 

2.468 

1275 

5 . 

1280 

60 

2.387 

2.127 

28 

2.155 

42 

2.637 

1408 

17 

1425 

98 

2.688 

1467 

20 

2.487 

no 

2.615 

2 370 

16 

1386 

38 

2.071 

1.894 

18 

1.912 

56 

2.888 

2.575 

35 

2.610 

55 

1083 

972 

11 

983 

45 

1.922 

1.720 

3 

1.723 

64 

1.786 

1.653 

2 

1.655 

118 

2.636 

2.361 

10 

1371 

87 

2.156 

1.935 

27 

1.962 

64 

2 899 

1616 

22 

2.638 

58 

2.744 

2.596 

6 

1602 

53 

1585 

2.287 

36 

1323 

88 

2.291 

2.049 

6 

1055 

74 

2.397 

2.109 

34 

1143 

55 

2.538 

1240 

60 

1300 

ne 

2.427 

2.219 

26 

2.245 

103 

2.968 

2.618 

7? 

2.697 

58 

2.145 

1.973 

28 

2.001 

114 

2.792 

1534 

11 

2.545 

84 

2.708 

2 400 

34 

2.434 

128 

2.795 

2.558 

?0 

2 578 

92 

2 351 

1143 

22 

2.165 

22 

2.886 

2 648 

48 

2.696 

Sec  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the 
derivation  of  weighted  participation.  For  the  national  sample.  Column  3 is  not  applicable. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-16 

Student  Participation  Rates,  Grade  8,  1996  Mathematics  Assessment,  Nonpublic  Schools 


Nor pubic 
School* 

Weighted 

Percentage 

Student 

Parfictpatton 

After 

Make-Up* 

Number  of  Number  of 

Student*  In  Student*  m 

Qritfnaf  Supplemental 
Samp*  Sample 

Number  of 
Student* 
Withdrawn 

Number  of 
Student* 
Excluded 

Number  of 
SkJdent* 

To  Be 
Assessed 

Number  of 
Students 
Assessed 
InffaJ 
Sessions 

Number  of 
Students 
Assessed 
Make-Up* 

Total  Number 
of  Students 
Assessed 

NATION 

97 

1.622 

7 

4 

1.611 

1.542 

14 

1.556 

JURISDICTIONS 

Alabama 

92 

130 

1 

3 

0 

128 

118 

1 

119 

Arkansas 

98 

66 

0 

2 

0 

64 

62 

0 

62 

California 

97 

237 

1 

0 

0 

238 

232 

0 

232 

Connecticut 

94 

282 

0 

2 

0 

280 

265 

0 

265 

Delaware 

96 

294 

0 

0 

0 

294 

281 

0 

281 

District  of  Columbia 

95 

237 

1 

4 

0 

234 

222 

0 

222 

Georgia 

97 

291 

8 

10 

6 

283 

267 

0 

267 

Guam 

95 

212 

1 

1 

0 

212 

199 

3 

202 

Iowa 

96 

295 

0 

0 

0 

295 

282 

0 

282 

Kentucky 

98 

228 

1 

5 

0 

224 

216 

0 

218 

Louisiana 

96 

455 

3 

7 

5 

446 

426 

0 

426 

Maryland 

97 

311 

0 

2 

1 

308 

296 

5 

301 

Massachusetts 

95 

314 

4 

2 

1 

315 

297 

4 

301 

Michigan 

96 

310 

1 

3 

0 

308 

293 

0 

293 

Minnesota 

96 

264 

2 

3 

0 

263 

250 

0 

250 

Missouri 

96 

371 

0 

3 

1 

367 

353 

0 

353 

Montano 

95 

125 

5 

1 

1 

128 

121 

0 

121 

Nebraska 

95 

376 

1 

2 

0 

375 

358 

0 

358 

Nevada 

95 

107 

1 

1 

0 

107 

101 

0 

101 

New  Hampshire 

96 

222 

2 

4 

1 

219 

212 

0 

212 

New  Jersey 

94 

353 

0 

c 

6 

342 

318 

2 

320 

New  Mexico 

89 

248 

5 

5 

0 

248 

228 

0 

228 

New  York 

95 

567 

4 

4 

0 

567 

539 

0 

539 

North  Dakota 

96 

199 

4 

1 

0 

202 

194 

0 

194 

Oregon 

93 

45 

1 

0 

0 

46 

43 

0 

43 

Rhod9  island 

96 

441 

2 

4 

0 

439 

423 

0 

423 

South  Carolina 

96 

168 

5 

3 

0 

170 

164 

0 

164 

Texas 

92 

180 

1 

1 

1 

179 

166 

0 

166 

Utah 

93 

44 

0 

0 

1 

43 

40 

0 

40 

Vermont 

95 

123 

0 

3 

0 

120 

114 

0 

114 

Washington 

97 

193 

0 

5 

0 

188 

182 

0 

132 

Wisconsin 

94 

384 

4 

3 

1 

384 

362 

0 

362 

Wyoming 

97 

53 

1 

1 

0 

53 

51 

0 

51 

See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  aid  for  the 
derivation  of  weighted  participation.  In  the  rows  for  national  sample,  the  dash  ( - ) means  “not  applicable.”  For  the 
national  sample,  Column  3 is  not  applicable. 

SOURCE:  National  Center  for  Education  Statistics.  National  Assessment  of  Educational  Progress  (NAEP).  1996 
Mathematics  Assessment. 
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Table  B-17 

Summary  of  School  and  Student  Participation,  Grade  8 
1996  Mathematics  Assessment,  Public  Schools 


Public 

Schools 

Weighted 

Percentage 

School 

Participation 

Before 

Substitution 

Notation 
Number  1 

Weighted 

Parcantaga 

School 

Participation 

After 

Substitution 

Notation 
Number  4 

Notation 
Number  6 

Weighted 
Percent  ege 
Student 
Participation 
After 

Make-Ups 

Notation 
Number  8 

Weighted 
Overall  Rati 

NATION 

81 

8? 

92 

76 

JURISDICTIONS 

Alabama 

84 

90 

93 

83 

Alaska 

92 

92 

80 

* 

74 

Arizona 

87 

87 

91 

79 

Arkansas 

70 

71 

• 

92 

65 

California 

83 

94 

90 

85 

Colorado 

100 

100 

91 

91 

Connecticut 

100 

100 

91 

91 

Doioware 

100 

100 

90 

90 

District  of  Columbia 

100 

'00 

85 

85 

DoDEA/DDESS 

100 

100 

98 

95 

DoDEA/DoDOS 

100 

100 

94 

94 

Florida 

100 

100 

91 

91 

Georgia 

99 

99 

90 

89 

Guam 

100 

100 

86 

86 

Hawaii 

100 

100 

91 

91 

Indiana 

88 

91 

93 

84 

iowa 

74 

64 

93 

79 

Kentucky 

88 

92 

94 

87 

Louisiana 

100 

100 

89 

89 

Maine 

90 

90 

92 

83 

Maryland 

86 

86 

91 

78 

Massachusetts 

92 

92 

92 

85 

Michigan 

70 

86 

• 

90 

77 

Minnesota 

86 

8S 

92 

81 

Mississippi 

89 

95 

93 

88 

Missouri 

93 

96 

91 

88 

Montana 

72 

75 

* 

9? 

69 

Nebraska 

99 

too 

91 

91 

Nevada 

38 

• 

38 

90 

34 

New  Hampshire 

66 

• 

69 

89 

62 

New  Jersey 

64 

• 

65 

V3 

60 

New  Mexico 

100 

100 

90 

90 

New  York 

71 

80 

• 

91 

72 

North  Carolina 

100 

'00 

91 

91 

North  Dakota 

83 

95 

94 

89 

Oregon 

86 

92 

90 

83 

Rhode  Island 

90 

90 

89 

80 

South  Carolina 

86 

87 

69 

7 7 

lennesseu 

92 

9? 

91 

83 

Texas 

90 

95 

V2 

88 

Utah 

100 

10C 

91 

91 

Vermont 

74 

74 

* 

93 

69 

Virginia 

100 

100 

91 

91 

Washington 

94 

95 

90 

86 

West  Virginia 

100 

'00 

92 

92 

Wisconsin 

78 

78 

• 

9? 

7? 

Wyoming 

100 

100 

93 

93 

See  preceding  lent  fur  explanations  of  the  nutations  and  guidelines  about  sample  representativeness  and  for  the  derivation  nl  weighted  participation 
Notation  Number  I The  Mate's  public  school  weighted  participation  rate  for  the  initial  sample  was  less  than  70% 

Notation  Numbci  4 The  stale's  public  school  weighted  participation  rate  fot  the  initial  sample  of  schools  was  below  85%  and  the  weighted  school 
participation  rate  after  substitution  was  below  90% 

Notation  Number  6.  The  nonparticipating  public  schools  included  a class  of  schools  with  similar  characteristics,  which  together  accounted  for  more  than  ^ 
ol  the  state's  total  fourth-grade  weighted  sample  of  public  schools 

Notation  Nutnhet  8 The  weighted  student  response  rate  within  participating  } Hie  schools  was  bdnw  v 

SOURCT  National  Center  tor  Kducntion  Statistics.  National  Assessment  of  I ducational  Progress  (NAP.P).  1 9%  Mathematics  Assessment 
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Table  B-18 

Summary  of  School  and  Student  Participation , Grade  8 
1 996  Mathematics  Assessment , Nonpublic  Schools 


Nonpublic 

Schools 

Weighted 

Percentage 

School 

Participation 

Before 

Substitution 

Notation 
Number  2 

Weighted 

Percentage 

School 

Participation 

After 

Substitution 

Notation 

Numbers 

Notation 
Number  7 

Weighted 

Percentage 

Student 

Participation 

After 

Make-Ups 

Notation 

Numbers 

Weighted 
Overall  Rate 

NATION 

81 

81 

97 

78 

JURISDICTIONS 

A.oDamo 

64 

• 

64 

92 

59 

A-kansas 

51 

* 

60 

98 

58 

California 

75 

75 

97 

73 

Connecticut 

63 

• 

65 

94 

61 

Delaware 

38 

• 

40 

96 

38 

District  Ot  Columbia 

47 

• 

47 

95 

45 

Georgia 

88 

88 

97 

86 

Guo  rr. 

76 

76 

95 

72 

Iowa 

88 

88 

96 

85 

Kentucky 

67 

• 

67 

98 

66 

Louisiana 

73 

73 

96 

70 

Maryland 

60 

• 

64 

97 

62 

Massachusetts 

'0 

7 4 

95 

70 

Michigan 

6C 

88 

96 

85 

Minnesota 

75 

75 

96 

72 

M»ssoun 

94 

iQO 

96 

96 

Montana 

70 

7* 

95 

75 

Nebraska 

83 

35 

95 

81 

Nevada 

78 

75 

pc 

73 

New  Hampshire 

85 

85 

• 

96 

82 

New  Jersey 

65 

* 

71 

• 

94 

67 

New  Mexico 

87 

8? 

89 

78 

New  Yonr 

88 

90 

• 

95 

66 

North  Dakota 

86 

86 

96 

82 

Oregon 

22 

22 

93 

2i 

Pnode  island 

8i 

?i 

96 

7S 

South  Comi  no 

76 

76 

96 

73 

Texas 

93 

93 

92 

86 

UlQh 

43 

* 

43 

93 

40 

Vermont 

73 

73 

95 

69 

Washington 

86 

86 

97 

Wisconsin 

68 

73 

94 

69 

W.cm-ng 

74 

* 

74 

97 

72 

See  preceding  text  for  explanations  of  the  notations  and  guidelines  about  sample  representativeness  and  for  the 
derivation  of  weighted  participation. 

Notation  Number  2:  The  state's  nonpuhlic  school  weighted  participation  rate  for  the  initial  sample  was  less  than  70%. 

Notation  Number  5:  The  state's  nonpublic  school  weighted  participation  rate  for  the  initial  sample  of  schools  was 
below  85 % and  the  weighted  school  participation  rate  after  substitution  was  below'  90%. 

Notation  Number  7:  The  nonparticipating  nonpublic  schools  included  a class  of  schools  w ith  similar  characteristics, 
which  together  accounted  for  more  than  5%  of  the  state's  total  fourth-grade  weighted  sample  of  nonpublic  schools. 

Notation  Number  9:  The  weighted  student  response  rate  within  participating  nonpublic  schools  was  below  859? . 

SOURCE:  National  Center  for  Education  Statistics.  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-19 

Weighted  Percentages  of  Students  Excluded  (SD  and  LEP)  from  Student  Sample,  Grade  8 
1996  Mathematics  Assessment,  Public  Schools 


Public 

School* 


NATION 

Northeast 

Southeast 

Central 

West 

JURISDICTIONS 


Total 

Percentage 
Student* 
Identified  SD 
and  LEP 


Total 

Percentage 

Students 

Excluded 


Percentage 
Student* 
Identified 
SD 


Alabama 

15 

8 

15 

Alaska 

15 

5 

10 

Arizona 

17 

9 

9 

Arkansas 

12 

7 

11 

California 

20 

10 

8 

Colorado 

11 

4 

10 

Connecticut 

16 

9 

14 

Delaware 

12 

8 

11 

District  of  Columbia 

12 

9 

9 

DoDEA/DDESS 

12 

4 

11 

DoDEA/DoDDS 

6 

2 

6 

Florida 

15 

9 

12 

Georgia 

9 

6 

8 

Guam 

9 

4 

6 

Hawaii 

13 

6 

10 

Indiana 

12 

5 

11 

Iowa 

11 

4 

10 

Kentucky 

10 

5 

10 

Louisiana 

9 

5 

8 

Maine 

10 

4 

9 

Maryland 

12 

6 

10 

Massachusetts 

15 

7 

14 

Michigan 

9 

5 

8 

Minnesota 

11 

3 

10 

Mississippi 

11 

7 

11 

Missouri 

12 

8 

12 

Montana 

10 

3 

9 

Nebraska 

11 

4 

10 

Nevada 

21 

10 

13 

New  Hampshire 

15 

4 

16 

New  Jersey 

12 

6 

9 

New  Mexico 

18 

8 

12 

New  York 

13 

7 

11 

North  Carolina 

9 

4 

8 

North  Dakota 

11 

4 

11 

Oregon 

12 

4 

11 

Rhode  island 

16 

7 

13 

South  Carolina 

10 

6 

9 

Tennessee 

11 

4 

1 1 

Texas 

16 

8 

11 

Utah 

12 

6 

10 

Vermont 

13 

5 

1? 

Virginia 

13 

7 

11 

Washington 

12 

5 

10 

West  Virginia 

12 

8 

12 

Wisconsin 

12 

7 

11 

Wyoming 

8 

1 

8 

Percentage 

Students 

Excluded 

SD 


Percentage 

Students 

Identified 

LEP 


Percentage 

Students 

Excluded 

LEP 


SD  = Students  with  Disabilities  (the  term  previously  used  was  1EP). 

LEP  = Limited  English  Proficienry.  . . 

To  be  excluded,  a student  was  supposed  to  be  classified  as  SD  or  as  LEP  and  judged  incapable  of  participating  in  the 
assessment.  A student  reported  as  belonging  to  both  SD  and  LEP  classifications  is  counted  once  in  the  overall  rate  (first 
column),  once  in  the  overall  excluded  rate  (second  column),  and  separately  in  the  remaining  columns. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP).  1996 
Mathematics  Assessment. 
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Table  B-20 

Weighted  Percentages  of  Students  Excluded  (SD  and  LEP)  from  Student  Sample , Grade  8 
} 996  Mathematics  Assessment,  Nonpublic  Schools 


Nonpublic 

Schools 

Total 

Percentage 
Student* 
Identified  SD 
and  LEP 

Total 

Percentage 

Students 

Excluded 

Percentage 

Student* 

Identified 

SD 

Percentage 

Student* 

Excluded 

SD 

Percentage 

Students 

Identified 

LEP 

Percentage 

Students 

Excluded 

LEP 

NATION 

3 

0 

2 

0 

1 

0 

JURISDICTIONS 

AJabama 

I 

c 

1 

c 

0 

0 

Arkansas 

0 

0 

0 

0 

0 

0 

California 

1 

0 

l 

0 

0 

0 

Connecticut 

5 

c 

4 

0 

1 

0 

Delaware 

2 

0 

2 

0 

0 

0 

District  of  Columbia 

i 

0 

0 

0 

1 

c 

Georgia 

3 

\ 

3 

1 

0 

0 

Guam 

1 

0 

0 

0 

1 

0 

Iowa 

2 

0 

? 

0 

0 

0 

Kentucky 

2 

0 

2 

0 

0 

0 

Louisiana 

A 

2 

4 

2 

0 

0 

Maryland 

1 

i 

\ 

1 

0 

0 

Massachusetts 

2 

s 

2 

1 

0 

0 

Michigan 

2 

0 

2 

c 

0 

0 

Minnesota 

2 

0 

1 

0 

1 

0 

Missouri 

3 

1 

3 

1 

0 

0 

Montana 

7 

1 

6 

1 

1 

0 

Nebraska 

1 

n 

1 

0 

0 

0 

Nevada 

0 

0 

0 

0 

0 

0 

New  Hampshire 

4 

1 

3 

0 

1 

1 

New  Jersey 

7 

3 

5 

} 

2 

2 

New  Mexico 

0 

0 

C 

0 

0 

0 

New  York 

0 

0 

0 

0 

0 

0 

North  Dakota 

2 

0 

? 

0 

0 

0 

Oregon 

0 

0 

0 

0 

0 

0 

Rhode  Island 

1 

0 

1 

0 

0 

0 

South  Carolina 

0 

0 

0 

0 

0 

0 

Texas 

1 

1 

\ 

1 

0 

0 

Utah 

4 

4 

4 

4 

0 

0 

Vermont 

3 

0 

3 

0 

0 

0 

Washington 

0 

0 

0 

0 

0 

0 

Wisconsin 

1 

0 

1 

0 

c 

0 

Wyoming 

6 

0 

6 

0 

0 

0 

SD  = Students  with  Disabilities  (the  term  previously  used  was  IEP). 

LEP  = Limited  English  Proficiency. 

To  be  excluded,  a student  was  supposed  to  be  classified  as  SD  or  as  LEP  and  judged  incapable  of  participating  in  the 
assessment.  A student  reported  as  belonging  to  both  SD  and  LEP  classifications  is  counted  once  in  the  overall  rate  (first 
column),  once  in  the  overall  excluded  rate  (second  column),  and  separately  in  the  remaining  columns. 

SOURCE:  National  Center  for  Education  Statistics,  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-21 

Weighted  Percentages  of  Absent,  SD,  and  LEP  Students  Based  on  Those  Invited  to  Participate 
Grade  8,  1996  Mathematics  Assessment,  Public  Schools 


Public 

Schools 

Weighted 

Percentage 

Student 

Participation 

After 

Make-Ups 

Weighted 

Percentage 

Absent 

Weighted 

Percentage 

Assessed 

SD 

Weighted 

Percentage 

Absent 

SD 

Weighted 

Percentage 

Assessed 

LEP 

Weighted 
Percentage 
Absent  LEP 

NATION 

92 

8 

84 

16 

96 

4 

JURISDICTIONS 

Alabama 

93 

7 

88 

12 

0 

100 

Alaska 

80 

20 

75 

25 

68 

32 

Anzona 

91 

9 

85 

15 

89 

11 

Arkansas 

92 

8 

86 

14 

100 

0 

California 

90 

10 

89 

1 1 

86 

14 

Colorado 

91 

9 

86 

14 

88 

12 

Connecticut 

91 

9 

84 

16 

75 

25 

Delaware 

90 

10 

81 

19 

65 

35 

District  of  Columbia 

85 

15 

76 

24 

86 

14 

DoDEA/DDESS 

95 

5 

92 

8 

0 

100 

DoDEA/DoDDS 

94 

6 

91 

9 

100 

0 

Florida 

91 

9 

86 

14 

100 

0 

Georgia 

90 

10 

89 

n 

82 

18 

Guam 

86 

14 

88 

12 

75 

25 

Hawaii 

91 

9 

78 

22 

92 

6 

Indiana 

93 

7 

85 

15 

100 

0 

Iowa 

93 

7 

85 

15 

100 

0 

Kentucky 

94 

6 

93 

7 

100 

0 

Louisiana 

89 

11 

82 

18 

100 

0 

Maine 

92 

8 

91 

9 

100 

0 

Maryland 

91 

9 

88 

12 

65 

35 

Massachusetts 

92 

8 

91 

9 

67 

33 

Michigan 

90 

10 

78 

22 

84 

16 

Minnesota 

92 

8 

89 

11 

87 

13 

Mississippi 

93 

7 

86 

14 

0 

100 

Missouri 

91 

9 

90 

10 

50 

50 

Montana 

92 

8 

91 

9 

28 

72 

Nebraska 

91 

9 

80 

12 

100 

0 

Nevada 

90 

10 

79 

21 

90 

10 

New  Hampshire 

89 

11 

84 

16 

100 

0 

New  Jersey 

93 

; 

86 

14 

87 

13 

New  Mexico 

90 

10 

80 

20 

96 

4 

New  York 

91 

9 

81 

19 

96 

4 

North  Carolina 

91 

9 

90 

10 

100 

0 

North  Dakota 

94 

6 

90 

10 

70 

22 

Oregon 

90 

10 

8? 

13 

100 

0 

Rhode  island 

89 

11 

86 

14 

95 

5 

South  Carolina 

89 

11 

85 

15 

100 

0 

Tennessee 

91 

9 

87 

13 

0 

100 

Texas 

92 

8 

82 

18 

92 

0 

Utah 

91 

9 

0? 

11 

100 

0 

Vermont 

93 

7 

90 

1C 

100 

C 

Virginia 

91 

9 

92 

8 

100 

0 

Washington 

90 

10 

87 

13 

95 

5 

West  Virginia 

92 

8 

87 

13 

0 

100 

Wisconsin 

92 

8 

91 

9 

78 

22 

Wyoming 

93 

7 

85 

15 

100 

0 

SD  = Students  with  Disabilities  (the  term  previously  used  was  IEP). 

LEP  = Limited  English  Proficiency. 

SOURCE:  National  Center  for  Education  Statistics.  National  Assessment  of  Educational  Piogress  (NAEP).  1996 
Mathematics  Assessment. 
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Table  B-22 

Weighted  Percentage  of  Absent,  SD,  and  LEP  Students  Bused  on  Those  Invited  to  Participate 
Grade  8,  1996  Mathematics  Assessment,  Nonpublic  Schools 


Nonpublic 

Schools 

Weighted 

Percentage 

Student 

Participation 

After 

Make-Ups 

Weighted 

Percentage 

Absent 

Weighted 

Percentage 

Assessed 

SD 

Weighted 

Percentage 

Absent 

SD 

Weighted 

Percentage 

Assessed 

LEP 

Weighted 

Percentage 

Absent 

LEP 

NATION 

97 

3 

100 

0 

100 

0 

JURISDICTIONS 

Alabama 

92 

8 

100 

0 

0 

mo 

Arkansas 

98 

2 

0 

100 

0 

100 

California 

97 

3 

100 

0 

0 

100 

Connecticut 

94 

6 

100 

0 

10C) 

0 

Delaware 

96 

4 

100 

0 

0 

100 

District  of  Columbia 

95 

5 

0 

100 

100 

0 

Georgia 

97 

3 

8V 

11 

0 

100 

Guam 

95 

5 

0 

100 

0 

100 

Iowa 

96 

4 

100 

u 

0 

100 

Kentucky 

98 

2 

100 

0 

0 

100 

Louisiana 

96 

4 

80 

20 

0 

100 

Maryland 

97 

3 

100 

0 

0 

100 

Massachusetts 

95 

5 

50 

50 

0 

100 

Michigan 

96 

4 

100 

0 

0 

100 

Minnesota 

96 

4 

100 

0 

ICO 

0 

Missouri 

96 

4 

100 

0 

0 

mo 

Montana 

95 

5 

100 

0 

100 

0 

Nebraska 

95 

5 

100 

0 

0 

100 

Nevada 

95 

5 

0 

100 

0 

100 

New  Hampshire 

96 

4 

66 

34 

0 

100 

New  Jersey 

94 

6 

100 

0 

0 

100 

New  Mexico 

89 

11 

0 

100 

0 

100 

New  York 

95 

5 

0 

100 

0 

100 

North  Dakota 

96 

4 

100 

0 

0 

100 

Oregon 

93 

7 

0 

100 

0 

100 

Rhode  island 

96 

4 

100 

0 

0 

100 

South  Carolina 

96 

4 

0 

100 

0 

100 

Texas 

92 

8 

0 

100 

0 

100 

Utah 

93 

7 

0 

100 

0 

100 

Vermont 

95 

5 

ion 

0 

0 

100 

Washington 

97 

3 

0 

100 

{) 

100 

Wisconsin 

94 

6 

100 

0 

0 

mo 

Wyoming 

97 

3 

100 

0 

0 

100 

SD  = Students  with  Disabilities  (the  term  previously  used  was  IEP). 

LEP  = Limited  English  Proficiency. 

SOURCE:  National  Center  for  Education  Statistics.  National  Assessment  ol  Educational  Progress  tNAEPt.  N% 
Mathematics  Assessment. 


Table  B-23 

Questionnaire  Response  Rates ; Grade  8 , /996  Mathematics  Assessment , Public  Schools 


Weighted  Percentage 

Percentage  of  School 

Weighted  Percentage 

of  Student*  Matched 

Characteristic*/ 

Percentage  of 

of  Student*  Matched 

to  School 

Policies 

SD/LEP  Student1 

Public 

to  Mathematic*  Teacher 

Characteristic*/ 

Questionnaires 

Questionnaires 

School* 

Questionnaires 

Policies  Questionnarie 

Returned 

Returned 

NATION 

86 

93 

89 

84 

JURISDICTIONS 

Alabama 

05 

90 

91 

88 

Alaska 

74 

76 

84 

64 

Arizona 

oo 

69 

88 

84 

Arkansas 

05 

87 

86 

94 

California 

00 

92 

93 

79 

Colorado 

80 

94 

94 

85 

Connecticut 

90 

61 

79 

89 

Delaware 

01 

94 

93 

92 

District  of  Columbia 

74 

77 

75 

66 

DoDEA/DDESS 

100 

83 

83 

96 

DoDEA/DoDDS 

05 

92 

95 

91 

Florida 

01 

96 

96 

86 

Georgia 

04 

88 

86 

91 

Guam 

56 

81 

83 

35 

Hawaii 

88 

97 

94 

87 

Indiana 

93 

93 

91 

87 

iowa 

95 

97 

98 

88 

Kentucky 

96 

93 

93 

91 

Louisiana 

00 

92 

90 

88 

Maine 

91 

93 

94 

88 

Maryland 

94 

82 

82 

88 

Massachusets 

92 

86 

85 

91 

Michigan 

02 

64 

84 

86 

Minnesota 

95 

85 

84 

91 

Mississippi 

92 

91 

91 

90 

Missouri 

92 

92 

92 

87 

Montana 

88 

89 

89 

90 

Nebraska 

95 

86 

90 

89 

Nevada 

88 

97 

96 

94 

New  Jersey 

96 

87 

87 

92 

New  Mexico 

97 

9‘ 

88 

85 

New  York 

B9 

81 

80 

62 

North  Carolina 

86 

76 

77 

79 

North  Dakota 

96 

9 1 

91 

89 

Oregon 

92 

94 

92 

84 

Pennsylvania 

87 

9? 

92 

79 

Rhode  Island 

94 

90 

88 

84 

South  Carolina 

92 

92 

91 

89 

Tennessee 

94 

95 

95 

92 

Texos 

97 

86 

86 

84 

Utah 

91 

91 

92 

83 

Vermont 

92 

92 

93 

89 

Virginia 

94 

97 

97 

82 

Washington 

89 

87 

87 

88 

West  Virginia 

96 

89 

90 

91 

Wisconsin 

88 

92 

91 

87 

Wyoming 

95 

94 

93 

88 

1 This  percentage  is  unweighted,  and  computed  over  all  students,  whereas  all  other  numbers  in  this  report  arc  based  on 
the  reporting  sample  only. 

SOURCE:  National  Center  for  Education  Statistics.  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Table  B-24 

Questionnaire  Response  Rates , Grade  8,  1996  Mathematics  Assessment,  Nonpuhlic  Schools 


Nonpublic 

Schools 

Weighted  Percentage 
of  Students  Matched 
to  Mathematic*  Teacher 
Questionnaire* 

Weighted  Percentage 
of  Students  Matched 
to  School 
Characteristic*/ 
Policies  Questlonnarie 

Percentage  of  School 
Characteristics/ 
Policies 
Questionnaires 
Returned 

Percentage  of 
SD/LEP  Student’ 
Questionnaire* 
Returned 

NATION 

91 

96 

93 

97 

JURISDICTIONS 

Alabama 

98 

93 

89 

IX 

Arkansas 

100 

100 

IX 

0 

California 

85 

84 

85 

IX 

Connecticut 

94 

80 

79 

IX 

Delaware 

100 

81 

83 

IX 

District  of  Columbia 

92 

81 

80 

86 

Georgia 

100 

100 

IX 

0 

Guam 

82 

76 

88 

IX 

Iowa 

99 

89 

87 

IX 

Kentucky 

91 

93 

90 

0 

Louisiana 

94 

86 

82 

IX 

Maryland 

95 

95 

94 

67 

Massachusetts 

92 

90 

88 

IX 

Michigan 

99 

89 

89 

IX 

Minnesota 

94 

100 

IX 

IX 

Missouri 

94 

70 

77 

IX 

Montana 

100 

86 

39 

88 

Nebraska 

68 

95 

95 

IX 

Nevada 

100 

73 

83 

0 

New  Hampshire 

92 

100 

IX 

IX 

New  Jersey 

90 

89 

91 

88 

New  Mexico 

100 

IX 

IX 

IX 

New  York 

94 

69 

67 

IX 

North  Dakota 

90 

90 

91 

90 

Oregon 

too 

IX 

IX 

0 

Rhode  Island 

99 

86 

88 

IX 

South  Carolina 

87 

92 

90 

IX 

Texas 

98 

IX 

IX 

IX 

Utah 

100 

IX 

IX 

IX 

Vermont 

100 

95 

89 

67 

Washington 

79 

IX 

IX 

0 

Wisconsin 

93 

98 

96 

IX 

Wyoming 

100 

IX 

IX 

IX 

1 This  percentage  is  unweighted,  and  computed  over  all  students,  whereas  all  other  numbers  in  this  report  are  based  on 
the  reporting  sample  only. 

SOURCE:  National  Center  for  Education  Statistics.  National  Assessment  of  Educational  Progress  (NAEP),  1996 
Mathematics  Assessment. 
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Appendix  C 


CONDITIONING  VARIABLES  AND  CONTRAST  CODINGS 


This  appendix  contains  information  about  the  conditioning  variables  used  in 
scaling/plausible  value  estimation  for  the  1996  State  Assessment  program.  The  initial  step  in 
construction  of  conditioning  variables  involves  forming  primary  student-based  vectors  of 
response  data  from  answers  to  student,  teacher,  and  school  questionnaires,  demographic  and 
background  data  such  as  supplied  by  Westat,  Inc.,  and  other  student  information  known  prior  to 
scaling.  The  initial  conditioning  vectors  concatenate  this  student  background  information  into  a 
series  of  identifying  “contrasts”  comprising: 

1 . Categorical  variables  derived  by  expanding  the  response  options  of  a 
questionnaire  variable  into  a binary  series  of  one-degree-of-freedom 
“dummy”  variables  or  contrasts,  (these  form  the  majority  of  each 
student  conditioning  vector); 

2.  Questionnaire  or  demographic  variables  that  possess  ordinal 
response  options,  such  as  number  of  hours  spent  watching  television, 
which  are  included  as  linear  and/or  quadratic  multi-degree-of- 
freedom  contrasts; 

3.  Continuous  variables,  such  as  student  logit  scores  based  on  percent 
correct  values,  included  as  contrasts  in  their  original  form  or  a 
transformation  of  their  original  form,  and; 

4.  Interactions  of  two  or  more  categorical  variables  forming  a set  of 
orthogonal  one-degree-of-freedom  dummy  variables  or  contrasts. 

This  appendix  gives  the  specifications  used  for  constructing  the  conditioning  variables. 
Table  C-l  defines  the  information  provided  for  each  main  sample  variable.  Table  C-2  provides  a 
summary  of  the  conditioning  variables  specifications  that  are  contained  in  the  remainder  of  this 
appendix. 

As  described  in  Chapter  9,  the  linear  conditioning  model  employed  for  the  estimation  of 
plausible  values  did  not  directly  use  the  conditioning  variable  specifications  listed  in  this 
appendix.  To  eliminate  inherent  instabilities  in  estimation  encountered  when  using  a large 
number  of  correlated  variables,  a principal  component  transformation  of  the  correlation  matrix 
obtained  from  the  conditioning  variable  contrasts  derived  according  to  these  primary 
specifications  was  performed.  The  principal  components  scores  based  on  this  transformation 
were  used  as  the  predictor  variables  in  estimating  the  linear  conditioning  model. 
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Table  C-l 

Description  of  Specifications  Provided  for  Each  Conditioning  Variable 


Title 

Description 

CONDITIONING  ID 

A unique  eight-character  ID  assigned  to  identify  each 
conditioning  variable  corresponding  to  a particular  background 
or  subject  area  question  within  the  entire  pool  of  conditioning 
variables.  The  first  four  characters  identify  the  origin  of  the 
variable:  BACK  (background  questionnaire),  SUBJ  (subject 
specific  questionnaire)  SCHL  (school  questionnaire),  TCHR 
(background  part  of  teacher  questionnaire),  and  TSUB  (subject 
classroom  pan  of  teacher  questionnaire).  The  second  four  digits 
represent  the  sequential  position  within  each  origin  group. 

DESCRIPTION 

A short  description  of  the  conditioning  variable. 

GRADES/ ASSESSMENTS 

Three  characters  identifying  assessment  (“S”  for  state,  “N”  for 
national;  and  grade  (04,  08,  and  12)  in  which  the  conditioning 
variable  was  used. 

CONDITIONING  VAR  LABEL 

A descriptive  eight-character  label  identifying  the  conditioning 
variable. 

NAEP  ID 

The  seven-character  NAEP  database  identification  for  the 
conditioning  variable. 

TYPE  OF  CONTRAST 

The  type  of  conditioning  variable.  “CLASS”  identifies  a 
categorical  conditioning  variable  and  "SCALE”  identifies 
continuous  or  quasi-continuous  conditioning  variables. 
“INTERACTION”  identifies  a set  of  orthogonal  contrasts 
formed  from  two  or  more  “CLASS”  variables.  “OTHER” 
conditioning  variables  do  not  fall  into  any  of  the  above  types. 

TOTAL  NUMBER  OF 
SPECIFIED  CONTRASTS 

Each  conditioning  variable  forms  a set  of  one  or  more  contrasts. 
For  each  valid  response  value  of  conditioning  variable  a 
contrast  must  be  defined.  One  or  more  response  values  may  be 
collapsed  together  to  form  one  contrast.  The  number  of 
response  value  “sets”  of  a conditioning  variable  forming  a 
unique  contrast  is  the  value  given  in  this  field. 

NUMBER  OF  INDEPENDENT 
CONTRASTS 

The  number  of  degree  of  freedom  in  a set  of  contrasts  formed 
from  a conditioning  variable.  For  a categorical  conditioning 
variable  this  number  would  be  the  number  of  response  options 
minus  one  if  each  response  option  formed  its  own  unique 
contrast. 

' * 
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Table  C-2 

Summary  Table  of  Conditioning  Variable  Specifications 
for  the  1996  State  Assessment  in  Mathematics 


Conditioning 

ID 

NAEP 

ID 

BKSER 

BACK0001 

DSEX 

BACK0002 

DRACE 

BACK0003 

B003I0I 

BACK0004 

TOL8 

BACK0005 

TOL5 

BACK0006 

DOC 

BACK0007 

PARED 

BACK0008 

SCHTYPE 

BACK0010 

IEP 

BACK001 1 

LEP 

BACK0012 

CHAP1 

BACK0013 

SLUNCH 

BACK0014 

B00320I 

BACK0015 

B008901 

BACK0016 

BOO 1801 

BACK0017 

BOO I 801 

BACK0018 

B006601 

BACK0019 

B006601 

BACK0020 

B006601 

BACK0021 

HOMEEN2 

BACK0022 

B005601 

BACK0023 

B005701 

BACK0024 

S004001 

BACK0025 

B008001 

BACK0026 

B007601 

BACK0027 

SCHNORM 

SCHL0001 

SCHNORM 

SCHL0002 

B007301 

BACK0029 

B007401 

BACK0030 

BOO 1101 

BACK0031 

B001 101 

BACK0032 

MA93FLG 

BACK0068 

MONSTUD 

BACK0069 

SUBS  AMP 

BACK0071 

RPTSAMP 

BACK0073 

B009301 

BACK0074 

B00940I 

BACK0075 

B009501 

BACK0076 

B009502 

BACK0077 

M8 12701 

SUBJ0002 

M8 12702 

SUBJ0003 

M8 12703 

SUBJ0004 

M8 12704 

SUBJ0005 

M8 12705 

SUBJ0006 

Grade  Grade 

Description 4 8 


GRAND  MEAN  (BOOK  COVER)  X X 

DERIVED  SEX  (WESTAT)  X X 

DERIVED  RACE  (WESTAT)  X X 

IF  HISPANIC,  WHAT  IS  YOUR  HISPANIC  BACKGROUND  X X 

TYPE  OF  LOCATION  (WESTAT)  (WESTAT)  X X 

TYPE  OF  LOCATION  (COLLAPSED)  (WESTAT)  X X 

DESCRIPTION  OF  COMMUNITY  (WESTAT)  X X 

PARENTS'  EDUCATION  (ETS)  X X 

SCHOOL  TYPE  (PQ)  X X 

INDIVIDUALIZED  EDUCATION  PLAN  (BOOK  COVER)  X X 

LIMITED  ENGLISH  PROFICIENCY  (BOOK  COVER)  X X 

X X 

SCHOOL  LUNCH  (WESTAT)  X X 

HOW  OFTEN  OTHER  THAN  ENGLISH  SPOKEN  IN  HOME  X X 

DO  YOU  HAVE  YOUR  OWN  STUDY  DESK  OR  TABLE  AT  XX 

HOW  MUCH  TELEVISION  DO  YOU  USUALLY  WATCH  EACH  X X 

HOW  MUCH  TELEVISION  DO  YOU  USUALLY  WATCH  EACH  X X 

TIME  SPENT  ON  HOMEWORK  X X 

TIME  SPENT  ON  HOMEWORK  X X 

TIME  SPENT  ON  HOMEWORK  X X 

HOME  ENVIRONMENT -ARTICLES  (OF  4)  IN  HOME  (ETS)  X X 

DOES  MOTHER  OR  STEPMOTHER  LIVE  AT  HOME  WITH  X X 

DOES  FATHER  OR  STEPFATHER  LIVE  AT  HOME  WITH  YOU  X X 

HOW  MANY  DAYS  OF  SCHOOL  MISSED  LAST  MONTH  X X 

HOW  LONG  LIVED  IN  THE  UNITED  STATES  X X 

HOW  MANY  GRADES  IN  THIS  STATE  (4)  X 

SCHOOL-LEVEL  NORMIT  GAUSSIAN  SCORE  (STUDENT)  X X 

SCHOOL-LEVEL  NORMIT  GAUSSIAN  SCORE  (STUDENT)  X X 

TIMES  CHANGED  SCHOOL  PAST  TWO  YEARS  X X 

DISCUSS  STUDIES  AT  HOME  X X 

HOW  MANY  PAGES  READ  IN  SCHOOL  AND  FOR  X X 

HOW  MANY  PAGES  READ  IN  SCHOOL  AND  FOR  X X 

METRO  ST  AT  AREA  (MSA)  6/30/93  DEF.  (WESTAT)  X X 

ACTUAL  MONITOR  STATUS  (STUDENT  LEVEL)  X X 

SAMPLE  TYPE  (S  I, S2, S3)  (ETS)  X X 

REPORTING  SAMPLE  ( 1 =YES,2=NO,0=EXCL  STUDENT)  X X 

HOW  OFTEN  USE  A HOME  COMPUTER  FOR  X X 

HOW  SAFE  DO  YOU  FEEL  AT  SCHOOL?  X X 

DOES  YOUR  STEP/MOTHER  WORK  AT  A JOB  FOR  PAY?  X 

DOES  YOUR  STEP/FATHER  WORK  AT  A JOB  FOR  PAY?  X 

HOW  OFTEN  DO  YOU  DO  MATH  PROBLEMS  FROM  X X 

HOW  OFTEN  DO  YOU  DO  MATH  PROBLEMS  ON  WORK  X X 

HOW  OFTEN  SOLVE  MATH  PROBLEMS  WITH  X X 

HOW  OFTEN  IN  MATH  WORK  W/  RULERS,  GEOM  SHAPES?  X 
HOW  OFTEN  WRITE  ABOUT  SOLVING  A MATH  PROBLEM?  X X 
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C037301  SCHL0023  SCHOOL  FOLLOW  DISTRICT/STATE  MATH  CURRICULUM?  X X 

C037302  SCHL0024  SCHOOL  FOLLOW  DISTRICT/STATE  SCIENCE  X X 

C037303  SCHL0025  SCHOOL  FOLLOW  DISTRICT/STATE  READING  X 

C037304  SCHL0026  SCHOOL  FOLLOW  DISTRICT/STATE  ARTS  CURRICULUM?  X X 

C037305  SCHL0027  SCHOOL  FOLLOW  DISTRICT/STATE  FOR  NONE  OF  ABOVE?  X X 

C037401  SCHL0028  SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  MATH?  X 

C037402  SCHL0029  SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  SCIENCE?  X 

C037403  SCHL0030  SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  READING?  X 

C037404  SCHL003 1 SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  ARTS?  X 

C037405  SCHL0032  SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  OTHER?  X 

C037406  SCHL0033  SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  NONE  X 

C037501  SCHL0034  4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  MATH?  X 

C037502  SCHL0035  4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  SCIENCE?  X 

C037503  SCHL0036  4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  READING?  X 

C037504  SCHL0037  4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  ARTS?  X 

C037505  SCHL0038  4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  NONE  OF  X 

C037601  SCHL0039  4TH  GRADERS  IN  SUMMER  PROGRAMS  IN  MATH?  X 

C037602  SCHL0040  4TH  GRADERS  IN  SUMMER  PROGRAMS  IN  SCIENCE?  X 

C037603  SCHL0041  4TH  GRADERS  IN  SUMMER  PROGRAMS  IN  READING?  X 

C037604  SCHL0042  4TH  GRADERS  IN  SUMMER  PROGRAMS  IN  ARTS?  X 

C037605  SCHL0043  4TH  GRADERS  IN  SUMMER  PROGRAMS  IN  NONE  OF  X 

C036601  SCHL0044  WHICH  BEST  DESCRIBES  PRIMARY  WAY  LIBRARY  X X 

C032207  SCHL0045  INVOLVE  PARENTS  AS  AIDES  IN  CLASSROOM?  X X 

C032209  SCHL0046  HAVE  PARENTS  REVIEW/SIGN  HOMEWORK?  X X 

C032210  SCHL0047  ASSIGN  HOMEWORK  STUDENTS  DO  WITH  PARENTS?  X X 

C032211  SCHL0048  HAVE  A PARENT  VOLUNTEER  PROGRAM?  X X 

C037701  SCHL0049  WHAT  % OF  PARENTS  IN  PARENT-TEACHER  ORGS?  X X 

C037702  SCHL0050  WHAT  % OF  PARENTS  IN  OPEN  HOUSE/BACK  SCHOOL  X X 

C037703  SCHL005 1 WHAT  % OF  PARENTS  IN  PARENT-TEACHER  X X 

C037704  SCHL0052  WHAT  % PARENTS  INVOLVED  MAKING  CURRICULUM  X X 

C037705  SCHL0053  WHAT  % OF  PARENTS  IN  VOLUNTEER  PROGRAMS?  X X 

C032402  SCHL0054  IS  STUDENT  ABSENTEEISM  A PROBLEM  IN  YOUR  X X 

C032401  SCHL0055  IS  STUDENT  TARDINESS  A PROBLEM  IN  YOUR  SCHOOL?  X X 

C032404  SCHL0056  ARE  PHYSICAL  CONFLICTS  A PROBLEM  IN  YOUR  X X 

C032406  SCHL0057  IS  TEACHER  ABSENTEEISM  A PROBLEM  IN  YOUR  X X 

C032407  SCHL0058  ARE  RACE/CULT.  CONFLICTS  A PROBLEM  IN  YOUR  a X 

0)32408  SCHL0059  IS  STUDENT  HEALTH  A PROBLEM  IN  YOUR  SCHOOL?  X X 

C032409  SCHL0060  IS  LACK  OF  PARENT  INVLMT  A PROBLEM  IN  YOUR  X X 

C032410  SCHL0061  IS  STUD  USE  OF  ALCOHOL  A PROBLEM  IN  YOUR  SCHOOL?  X X 

C03241 1 SCHL.0062  IS  STUDENT  TOBACCO  USE  A PROBLEM  IN  YOUR  X X 

C032412  SCHL0063  IS  STUDENT  DRUG  USE  A PROBLEM  IN  YOUR  SCHOOL?  X X 

C032413  SCHL0064  ARE  GANG  ACTIVITIES  A PROBLEM  IN  YOUR  SCHOOL?  X X 

C032414  SCHL0065  IS  STUDENT  MISBEHAVIOR  A PROBLEM  IN  YOUR  X X 

C0324 1 5 SCHL0066  IS  STUDENT  CHEATING  A PROBLEM  IN  YOUR  SCHOOL?  X X 

C032502  SCHL0067  IS  TEACHER  MORALE  POS.  OR  NEG.? X X 


285 


' ' n o 

J 


Table  C-2  (continued) 

Summary  Table  of  Conditioning  Variable  Specifications 
for  the  1996  State  Assessment  in  Mathematics 


Conditioning  NAEP 

ID ID Description 

C032503  SCHL0068  ARE  STUDENT  ATTITUDES  TO  ACADEMICS  POS.  OR  NEG.? 

C032505  SCHL0069  IS  PARENT  SUPPORT  FOR  ACHIEVEMENT  POS.  OR  NEG.? 

C032506  SCHL0070  IS  REGARD  FOR  SCHOOL  PROPERTY  POS.  OR  NEG.? 

C03360 1 SCHL007 1 % ABSENT  ON  AVERAGE  DAY? 

C03650 1 SCHL0072  WHAT  % OF  TEACHERS  ABSENT  ON  GIVEN  DAY? 

C03780 1 SCHL0073  % OF  STUDS  EROLLED  AT  START  OF  YR  EROLLED  AT 

C03790 1 SCHL0074  % OF  4TH  GRADERS  HELD  BACK  & REPEATING  4TH 

C03800 1 SCHL0075  % OF  FULL  TIME  TEACHERS  LEFT  BEFORE  END  OF  YR? 

C03830 1 SCHL0076  IS  SCHOOL  IN  NATIONAL  SCHOOL  LUNCH  PROGRAM? 

C03880 1 SCHL0077  SCHOOL  RECEIVE  CHAP  1 /TITLE  1 FUNDING? 

C034 1 0 1 SCHL0078  DID  PRINCIPAL  FILL  OUT  THIS  QUESTIONNAIRE 

C034 1 02  SCHL0079  DID  HEADMASTER/HEADMISTRESS  FILL  OUT 

C034 1 03  SCHL0080  DID  HEAD  TEACHER  FILL  OUT  THIS  QUESTIONNAIRE 

C034 1 04  SCHL008 1 DID  VICE  PRINCIPAL  FILL  OUT  THIS  QUESTIONNAIRE 

C034 1 05  SCHL0082  DID  COUNSELOR  FILL  OUT  THIS  QUESTIONNAIRE 

C034 1 06  SCHL0083  DID  CURRICULUM  COORD  FILL  OUT  THIS 

C034 1 07  SCHL0084  DID  TEACHER  FILL  OUT  THIS  QUESTIONNAIRE 

C034 1 08  SCHL0085  DID  SECRETARY  FILL  OUT  THIS  QUESTIONNAIRE 

C034 1 09  SCHL0086  DID  OTHER  PERSON  FILL  OUT  THIS  QUESTIONNAIRE 

T05  590 1 TCHR000 1 WHAT  IS  YOUR  GENDER? 

T05600I  TCHR0002  WHICH  BEST  DESRIBES  YOU? 

T040301  TCHR0003  YEARS  TAUGHT 

T056101  TCHR0004  HOW  MANY  YRS  TOTAL  YOU  TAUGHT  MATH? 

T056 102  TCHR0005  HOW  MANY  YRS  TOTAL  YOU  TAUGHT  SCIENCE? 

T05620 1 TCHR0006  TYPE  TCHNG  CERT  IN  THIS  ST  IN  MAIN  FIELD? 

T04050 1 TCHR0007  CERTIFICATION,  ELEMENTARY  OR  MIDDLE/JUNIOR  HS 

T040506  TCHR0008  DO  YOU  HAVE  CERTIFICATION  IN  ELEMENTARY  MATH? 

T040504  TCHR0009  DO  YOU  HAVE  CERTIFICATION  IN  JR  HIGH/SEC  MATH? 

T040505  TCHR0012  CERTIFICATION,  OTHER 

T05630 1 TCHR001 3 HIGHEST  ACADEMIC  DEGREE  YOU  HOLD? 

T040701  TCHR0014  EDUCATION  UNDERGRAD  MAJOR 

T040706  TCHR001 5 ELMENT  ED  UNDERGRAD  MAJOR 

T040707  TCHR0016  SEC  ED  UNDERGRAD  MAJOR 

T040703  TCHR001 7 WAS  YOUR  UNDERGRADUATE  MAJOR  MATH? 

T040704  TCHR00 1 8 WAS  YOUR  UNDERGRADUATE  MAJOR  MATH  ED? 

T040708  TCHR0023  SPECIAL  EDUCATION  UNDERGRAD  MAJOR 

T040709  TCHR0024  BILINGUAL  ED/ESL  UNDERGRAD  MAJOR 

T040705  TCHR0025  OTHER  UNDERGRAD  MAJOR 

T040801  TCHR0026  EDUCATION  GRAD  MAJOR 

T040807  TCHR0027  ELEMENTARY  ED  GRAD  MAJOR 

T040808  TCHR0028  SECONDARY  ED  GRAD  MAJOR 

T040803  TCHR0029  WAS  YOUR  GRADUATE  MAJOR  MATHEMATICS? 

T040804  TCHR0030  WAS  YOUR  GRADUATE  MAJOR  MATH  ED? 

T040809  TCHR0035  SPECIAL  ED  GRAD  MAJOR 

T04O8IO  TCHR0036  BILINGUAL  GRAD  MAJOR 


Grade  Grade 
4 8 

X X 

X X 

X X 

X X 

X X 

X X 

X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X 
X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 

X X 
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Table  C-2  (continued) 

Summary  Table  of  Conditioning  Variable  Specifications 
for  the  1996  State  Assessment  in  Mathematics 


Conditioning 

ID 

T040811 

T040812 

T040S13 

T040805 

T040806 

T056401 

T056402 

T056403 

T056404 

T056405 

T056406 

T056407 

T056408 

T056409 

T056410 

T056411 

T056412 

T056501 

T056601 

T056701 

T056702 

T056703 

T056704 

T056705 

T056706 

T056707 

T056708 

T056709 

T056710 

T0567 1 1 

T056712 

T056713 

T041201 

T041302 

T041303 

T056801 

T056901 

T0569A1 

T0569B1 

T0569C1 

T056902 

T0569A2 

T0569B2 

T0569C2 

7056903 


NAEP  Grade 

ID  Description  4 


TCHR0037  ADMIN/SUPERVISION  GRAD  MAJOR  X 

TCHR0038  CURRICULUM/INSTRUCTION  GRAD  MAJOR?  X 

TCHR0039  COUNSELING  GRAD  MAJOR?  X 

TCHR0040  OTHER  GRAD  MAJOR  X 

TCHR0041  NO  GRADUATE  STUDY  X 

TCHR0042  UNDERGRAD/GRAD  MINOR  STUDY-EDUCATION  X 

TCHR0043  UNDERGRAD/GRAD  MINOR  STUDY-ELEMENTARY  ED  X 

TCHR0044  UNDERGRAD/GRAD  MINOR  STUDY-SECONDARY  ED  X 

TCHR0045  UNDERGRAD/GRAD  MINOR  STUDY -MATHEMATICS  X 

TCHR0046  UNDERGRAD/GRAD  MINOR  STUDY-MATHEMATICS  ED  X 

TCHR005 1 UNDERGRAD/GRAD  MINOR  STUDY-SPECIAL  ED  X 

TCHR0052  UNDERGRAD/GRAD  MINOR  STUDY-BILINGUAL  ED  X 

TCHR0053  UNDERGRAD/GRAD  MINOR  STUDY-ADMIN  & X 

TCHR0054  UNDERGRAD/GRAD  MINOR  STUDY-CURRICULUM  & X 

TCHR0055  UNDERGRAD/GRAD  MINOR  STUDY-COUNSELING  X 

TCHR0056  UNDERGRAD/GRAD  MINOR  STUDY-OTHER  X 

TCHR0057  UNDERGRAD/GRAD  MINOR  STUDY-NONE  X 

TCHR0058  LAST  YR,  HOW  MUCH  TIME  IN  MATH/MATH  ED  X 

TCHR0060  LAST  2 YRS,  HOW  MANY  MATH/MATH  ED  UNIV  COURSES?  X 

TCHR0062  PAST  5 YRS,  TAKEN  COURSES/IN  PRO  DEVP-TELECOMM  X 

TCHR0063  PAST  5 YRS,  TAKEN  COURSES/IN  PRO  DEVP-TECH  USE  X 

TCHR0064  PAST  5 YRS,  TAKEN  COURSES/IN  PRO  DEVP-COOP  X 

TCHR0065  PAST  5 YRS,  COURSES/IN  PRO  DEVLP-INTERDISP INSTRCT  X 

TCHR0066  PAST  5 YRS,  COURSES/IN  PRO  DEVLP-PORTFOLIO  ASSMNT  X 

TCHR0067  PAST  5 YRS,  COURSES/IN  PRO  DEVLP-PERF  BASED  X 

TCHR0068  PAST  5 YRS,  COURSES/PRO  DEVLP-TEACH  HIGHORDER  X 

TCHR0069  PAST  5 YRS,  COURSES/PRO  DEVLP-TEACH  DIFF  CULT  X 

TCHR0070  PAST  5 YRS,  COURSES/PRO  DEVLP-TEACH  LEP  STUDENTS  X 

TCHR007 1 PAST  5 YRS,  COURSES/PRO  DEVLP-TEACH  SPEC  NEED  X 

TCHR0072  PAST  5 YRS,  COURSES/PRO  DEVLP-CLASSRM  MNGMT/ORG  X 

TCHR0073  PAST  5 YRS,  COURSES/PRO  DEVLP-OTHER  PROF  ISSUES  X 

TCHR0074  PAST  5 YRS,  COURSES/PRO  DEVLP-NONE  OF  ABOVE  X 

TCHR0075  AVAILABILITY  OF  RESOURCES  X 

TCHR0076  ARE  CURRICULUM  SPECIALISTS  AVAILABLE  FOR  MATH?  X 

TCHR0077  SCIENCE  CURRICULUM  SPECIALIST  X 

TCHR0078  HOW  MANY  SCHOOL  HOURS  ARE  PREP  TIME  PER  WEEK?  X 

TCHR0079  METHODS  OF  TEACHING  ELEM  MATH-  1 +COLLEGE  X 

TCHR0080  METHODS  OF  TEACHING  ELEM  MATH-PART  COLLEGE  X 

TCHR008 1 METHODS  OF  TEACHING  ELEM  MATH-SEMINAR  X 

TCHR0082  METHODS  OF  TEACHING  ELEM  MATH-LITTLE  NO  X 

TCHR0083  NUMBER  SYSTEMS  & NUMERATION- 1 + COLLEGE  COURSE  X 

TCHR0084  NUMBER  SYSTEMS  & NUMERATION-PART  COLLEGE  X 

TCHR0085  NUMBER  SYSTEMS  & NUMERATION-SEMINAR  X 

TCHR0086  NUMBER  SYSTEMS  & NUMERATION-LITTLE/NO  X 

TCHR0087  MEASUREMENT  IN  MATH-  1 +COLLEGE  COURSE X 


Grade 

8 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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X 

X 

X 
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X 

X 

X 

X 

X 

X 

X 

X 
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Table  C-2  (continued) 

Summary  Table  of  Conditioning  Variable  Specifications 
for  the  1996  State  Assessment  in  Mathematics 


Conditioning 

ID 

NAEP 

ID 

Description 

Grade 

4 

Grade 

8 

T0569A3 

7CHR0088 

MEASUREMENT  IN  MA7H-  PAR7  COLLEGE  COURSE 

X 

X 

T0569B3 

7CHR0089 

MEASUREMENT  IN  MATH-  -SEMINAR 

X 

X 

T0569C3 

7CHR0090 

MEASUREMENT  IN  MATH-  -LITTLE  NO  EXPOSURE 

X 

X 

T056904 

7CHR009 1 

GEOMETRY-  1 +COLLEGE  COURSE 

X 

X 

T0569A4 

7CHR0092 

GEOMETRY-PART  COLLEGE  COURSE 

X 

X 

T0569B4 

7CHR0093 

GEOMETRY-SEMINAR 

X 

X 

T0569C4 

7CHR0094 

GEOMETRY-LITTLE  NO  EXPOSURE 

X 

X 

T056905 

7CHR0095 

PROBABILITY/STATISTICS-  1+COLLEGE  COURSE 

X 

X 

T0569A5 

7CHR0096 

PROBABILITY/STATISTICS-PART  COLLEGE  COURSE 

X 

X 

T0569B5 

7CHR0097 

PROBABILITY/STATISTICS-SEMINAR 

X 

X 

T0569C5 

7CHR0098 

PROBABILITY/STATISTICS-LITTLE  NO  EXPOSURE 

X 

X 

T056906 

7CHR0099 

COLLEGE  ALGEBRA-  1 +COLLEGE  COURSE 

X 

X 

T0569A6 

7CHR0100 

COLLEGE  ALGEBRA-PART  COLLEGE  COURSE 

X 

X 

T0569B6 

7CHR0101 

COLLEGE  ALGEBRA-SEMINAR 

X 

X 

T0569C6 

7CHR0102 

COLLEGE  ALGEBRA-LITTLE  NO  EXPOSURE 

X 

X 

T056907 

7CHR0103 

CALCULUS-  1+COLLEGE  COURSE 

X 

X 

T0569A7 

7CHR0I04 

CALCULUS-PART  COLLEGE  COURSE 

X 

X 

T0569B7 

7CHR0105 

CALCULUS-SEMINAR 

X 

X 

T0569C7 

7CHR0106 

CALCULUS-LITTLE  NO  EXPOSURE 

X 

X 

T056908 

7CHR0107 

ABSTRACT/LINEAR  ALGEBRA-  1+COLLEGE  COURSE 

X 

X 

T0569A8 

TCHR0108 

ABSTRACT/LINEAR  ALGEBRA-PART  COLLEGE  COURSE 

X 

X 

T0569B8 

7CHR0109 

ABSTRACT/LINEAR  ALGEBRA-SEMINAR 

X 

X 

T0569C8 

7CHR01 10 

ABSTRACT/LINEAR  ALGEBRA-LITTLE  NO  EXPOSURE 

X 

X 

T057001 

7CHR01 1 1 

EVER  STUDIED  ESTIMATION? 

X 

X 

7057002 

TCHR01 1 2 

EVER  STUDIED  PROBLEM  SOLVING  IN  MATH? 

X 

X 

T057003 

TCHR0113 

EVER  STUDIED  USE  OF  MANIPULATIVES? 

X 

X 

T057004 

TCHR01 14 

EVER  STUDIED  USE  OF  CALCULATORS  IN  MATH  INSTRC? 

X 

X 

T057005 

7CHR01 15 

EVER  STUDIED  UNDERSTANDING  STUDS  MATH 

X 

X 

T057006 

TCHR01 16 

EVER  STUDIED  GENDER  ISSUES  IN  TEACHING  MATH? 

X 

X 

T057007 

TCHR0II7 

EVER  STUDIED  TEACHING  STUDS  OF  DIFF  CULTURES? 

X 

X 

T05710I 

TCHR01 18 

KNOWLEDGE  OF  NCTM  CURR  & EVAL  STANDARDS  FOR 

X 

X 

77)5720 1 

TCHR01 19 

PRO  ACTVTS-STRATEGIES  LOCAL  WORKSHOPS 

X 

X 

T05721 1 

7CHR0120 

PRO  ACTVTS-STRATEGIES  REGIONAL  NCTM  MEETING 

X 

X 

TOS722I 

7CHR0121 

PRO  ACTVTS-STRATEGIES  NATIONAL  NCTM  MEETING 

X 

X 

T057231 

TCHR0122 

PRO  ACTVTS-STRATEGIES  OTHER 

X 

X 

T057241 

TCHR0123 

PRO  ACTVTS-STRATEGIES  NO 

X 

X 

7*057301 

TSUBOOOl 

IMPORTANCE  W/  STUDS-APPLYING  MATH  IDEAS? 

X 

X 

T057302 

TSUB(KX)2 

IMPORTANCE  W/  STUDS-PROB  SOLVING=GOAL  & 

X 

X 

T057303 

TSUB0003 

IMPORTANCE  W / STUDS-?  TECHS  PROMOTE  STUD  TALK? 

X 

X 

T( 157304 

TSUB0004 

IMPORTANCE  W / STUDS-USE  RESULTS  TO  INFORM 

X 

X 

77)5740! 

TSUB0005 

TO  ACCESS  PROGRESS  HOW  OFTEN  USE  MULT  CHOICE 

X 

X 

T057402 

7SUB0006 

TO  ACCESS  PROGRESS  HOW  OFTEN  USE  PROBLEM  SETS 

X 

X 

T(  157403 

7SUB0007 

TO  ACCESS  PROGRESS  HOW  OFTEN  USE  SHORT  WRITTEN 

X 

X 

7057 404 

TSUB0008 

TO  ACCESS  PROGRESS  HOW  OFTEN  USE  INDV/GROUP 

X 

X 

'17)57405 

TSUB0009 

TO  ACCESS  PROGRESS  HOW  OFTEN  USE  PORTFOLIOS 

X 

X 

2XX 


r 


•J 


' » 


t 
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T057501  TSUB0010  BEST  DESCRIPTION  OF  COMPUTER  AVAILABLITY  IN  X X 

T057601  TSUB001 1 PRIMARY  USE  OF  COMPUTERS  FOR  MATH  INSTRUCTION?  X X 

T044002  TSUB0012  ARE  STUDENTS  ASSIGNED  TO  THIS  CLASS  BY  ABILITY?  X X 

T057701  TSUB0013  IF  ASSIGNED  BY  ABILITY,  WHAT  BASIS  ASSIGNED?  X X 

T057801  TSUB0014  IF  ASSIGNED  BY  ABILITY,  WHAT  IS  MATH  ABILITY?  X X 

T04420I  TSUB0015  CREATE  GROUPS  IN  CLASS  FOR  MATH  ON  ABILITY  BASIS?  X X 

T044301  TSUB0016  TIMEAVEEK  ON  MATH  INSTRUTION?  X X 

TC  .17901  TSUB0017  HOW  MUCH  TIME  PER  WEEK  STUDS  DO  MATH  W/  X X 

T044401  TSUB0018  AMOUNT  MATH  HOMEWORK  ASSIGN/DAY?  X X 

T044501  TSUB0019  HOW  OFTEN  DO  STUDENTS  DO  MATH  FROM  TEXTBOOKS?  X X 

T044502  TSUB0020  HOW  OFTEN  DO  STUDENTS  DO  MATH  ON  WORKSHEETS?  X X 

T044512  TSUB0021  HOW  OFTEN  DO  STUDENTS  SOLVE  PROBS  W/ OTHER  X X 

T044513  TSUB0022  HOW  OFTEN  DO  STUDENTS  WORK  W/ OBJECTS  LIKE  X X 

T044514  TSUB0023  HOW  OFTEN  WORK  W/ COUNTING  BLOCKS. GEOMETRIC  X X 

T044505  TSUB0024  HOW  OFTEN  DO  STUDENTS  USE  A CALCULATOR?  X X 

T044515  TSUB0025  HOW  OFTEN  DO  STUDENTS  TAKE  MATH  TESTS?  X X 

T044507  TSUB0026  HOW  OFTEN  DO  STUDENTS  WRITE  ABOUT  PROBLEM-  X X 

T044516  TSUB0027  HOW  OFTEN  DO  STUDENTS  TALK  ABOUT  MATH  WORK?  X X 

T044508  TSUB0028  HOW  OFTEN  DO  STUDENTS  WRITE  REPORTS/DO  X X 

T044509  TSUB0029  HOW  OFTEN  DO  STUDENTS  DISCUSS  MATH  W/OTHER  X X 

T044510  TSUB0030  HOW  OFTEN  DO  STUDENTS  WORK  REAL-LIFE  MATH  X X 

T044506  TSUB0031  HOW  OFTEN  DO  STUDENTS  USE  A COMPUTER?  X X 

T058001  TSUB0032  IN  MATH  CLASS  HOW  OFTEN  ADDRESS-NUMBERS  & OPS?  X X 

T058002  TSUB0033  IN  MATH  CLASS  HOW  OFTEN  ADDRESS-MEASUREMENT?  X X 

T058003  TSUB0034  IN  MATH  CLASS  HOW  OFTEN  ADDRESS-GEOMETRY?  X X 

T058004  TSUB0035  IN  MATH  CLASS  HOW  OFTEN  ADDRESS-DATA  ANALYSIS?  X X 

T058005  TSUB0036  IN  MATH  CLASS  HOW  OFTEN  ADDRESS-ALGEBRA  & X X 

T058006  TSUB0037  IN  MATH  HOW  OFTEN  ADDRESS-LRN  MATH  X X 

T058007  TSUB0038  IN  MATH  HOW  OFTEN  ADDRESS-LRN  X X 

T058008  TSUB0039  IN  MATH  HOW  OFTEN  ADDRESS-DEVELOP  REASONING  X X 

T058009  TSUB0Q40  IN  MATH  HOW  OFTEN  ADDRESS-LRN  TO  COMMUNICATE  X X 

T045401  TSUB0041  DO  YOU  PERMIT  UNRESTRICTED  USE  OF  CALCULATORS?  X X 

T044801  TSUB0042  DO  YOU  PERMIT  USE  OF  CALCULATORS  ON  TESTS?  X X 

T045001  TSUB0043  STUDENTS  HAVE  ACCESS  TO  SCHL-OWNED  X X 

T044901  TSUB0044  DO  YOU  PROVIDE  INSTRUCTION  IN  USE  OF  X X 

T045304  TSUB0045  HOW  PREPARED  TO  TEACH  MATH  CONCEPTS?  X X 

T045305  TSUB0046  HOW  PREPARED  TO  TEACH  MATH  PROCEDURES?  X X 

T045302  TSUB0047  HOW  PREPARED  TO  TEACH  USE  OF  COMPUTERS?  X X 

T045303  TSUB0048  HOW  PREPARED  TO  TEACH  USE  OF  CALCULATORS?  X X 

T044000  TSUB0049  NUMBER  OF  STUDENTS  IN  CLASS?  X X 

NTLUNSC  SCHL0173  PERCENT  IN  NATIONAL  LUNCH  PROGRAM  FOR  SCHOOL  X X 

NTLUNSC  SCHL0174  PERCENT  xN  NATIONAL  LUNCH  PROGRAM  FOR  SCHOOL  X X 

REMRDSC  SCHL0175  PERCENT  IN  REMEDIAL  READING  PRGRM  FOR  SCHOOL  X X 

REMRDSC  SCHL0176  PERCENT  IN  REMEDIAL  READING  PRGRM  FOR  SCHOOL  X X 

REMMHSC  SCHL0177  PERCENT  IN  REMEDIAL  MATH  PROGRAM  FOR  SCHOOL  X X 
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Table  C-2  (continued) 

Summary  Table  of  Conditioning  Variable  Specifications 
for  the  1996  State  Assessment  in  Mathematics 

Conditioning  NAEP  Grade  Grade 

IP ID Description 4 8 

REMMHSC  SCHL0178  PERCENT  IN  REMEDIAL  MATH  PROGRAM  FOR  SCHOOL  X X 

NTLUNGR  SCHL0179  PERCENT  IN  NATIONAL  LUNCH  PROGRAM  FOR  GRADE  X 

NTLUNGR  SCHL0I80  PERCENT  IN  NATIONAL  LUNCH  PROGRAM  FOR  GRADE  X 

REMRDGR  SCHL0181  PERCENT  IN  REMEDIAL  READING  PRGRM  FOR  GRADE  X 

REMRDGR  SCHL0182  PERCENT  IN  REMEDIAL  READING  PRGRM  FOR  GRADE  X 

REMMHGR  SCHL0183  PERCENT  IN  REMEDIAL  MATH  PROGRAM  FOR  GRADE  X 

REMMHGR  SCHL0184  PERCENT  IN  REMEDIAL  MATH  PROGRAM  FOR  GRADE  X 

MATTAKE  SUBJ0001  MATH  CLASS  TAKING  THIS  YEAR  (ETS)  - X 

B 009701  BACK0078  DESCRIBE  YOUR  OVERALL  GRADES  SINCE  6TH  GRADE  - X 

B009801  BACK0079  HOW  FAR  IN  SCHOOL  DO  YOU  THINK  YOU  WILL  GO?  - X 

B009601  BACK0080  DOES  YOUR  STEP/MOTHER  WORK  AT  A JOB  FOR  PAY?  - X 

B 009602  BACK0081  DOES  YOUR  STEP/FATHER  WORK  AT  A JOB  FOR  PAY?  - X 

M812712  SUBJ0032  WORK  W/ MEAS.  INSTRUMENTS/GEOM  SOLIDS  FOR  - X 

M812713  SUBJ0033  HOW  OFTEN  WRITE  REPORTS  OR  DO  MATH  PROJECTS?  - X 

M8 12201  SUBJ0034  IS  THERE  A PORTFOLIO  W/ YOUR  MATH  WORK  IN  IT?  - X 

M8 12301  SUBJ0035  DO  YOU  USE  A SCIENTIFIC  CALCULATOR  FOR  MATH  - X 

M8 12401  SUBJ0036  DO  YOU  USE  A GRAPHING  CALCULATOR  FOR  MATH  - X 

MATEXP  SUBJ0037  MATH  CLASS  EXPECT  TO  TAKE  IN  NINTH  GRADE  (ETS)  - X 

M810701  SUBJ0038  DO  YOU  AGREE:  I LIKE  MATH  - X 

M8 10703  SUBJ0039  DO  YOU  AGREE:  I AM  GOOD  IN  MATH  - X 

M8 10707  SUBJ0040  AGREE/DISAGREE:  UNDERSTAND  MOST  OF  MATH  CLASS  - X 

M810709  SUBJ0041  AGREE/DISAGREE:  ONLY  ONE  WAY  TO  SOLVE  MATH  - X 

M8 10708  SUBJ0042  AGREE/DISAGREE:  MATH  IS  MOSTLY  MEMORIZING  FACTS  - X 

M810710  SUBJ0043  AGREE/DISAGREE:  CONCEPTS  ARE  AS  IMPORTANT  AS  - X 

M8 10705  SUBJ0044  DO  YOU  AGREE:  MATH  USEFUL/SOLVING  EVERYDAY  - X 

M8 10706  SUBJ0045  AGREE/DISAGREE:  WOULD  NOT  STUDY  MORE  MATH  - X 

M810711  SUBJ0046  AGREE/DISAGREE:  ALL  CAN  DO  WELL  IN  MATH  IF  TRY  - X 

C034201  SCHL0087  BEST  DESCRIBES  HOW  8TH  GRADES  ARE  ORGANIZED?  - X 

C034402  SCHL0088  ARE  8TH-GRADERS  ASSIGNED  TO  MATH  BY  ABILITY?  - X 

C034403  SCHL0089  ARE  8TH-GRADERS  ASSIGNED  TO  SCIENCE  BY  ABILITY?  - X 

C034401  SCHL0090  ARE  8TH-GRADERS  ASSIGNED  TO  ENGLISH  BY  ABILITY?  - X 

C034406  SCHL0091  ARE  8TH-GRADERS  ASSIGNED  TO  ARTS  BY  ABILITY?  - X 

C034510  SCHL0092  HOW  OFTEN  8TH  GRDS  RECEIVE  COMP  SCI  INSTRUCTION?  - X 

C034511  SCHL0093  HOW  OFTEN  8TH  GRDS  RECEIVE  MATH  INSTRUCTION?  - X 

C034512  SCHL0094  HOW  OFTEN  8TH  GRDS  RECEIVE  SCIENCE  INSTRUCTION?  - X 

C034513  SCHL0095  HOW  OFTEN  8TH  GRDS  RECEIVE  ENGLISH  INSTRUCTION?  - X 

C034514  SCHL0096  HOW  OFTEN  8TH  GRDS  RECEIVE  ARTS  INSTRUCTION?  - X 

C03161I  SCHL0097  HAS  ENGLISH  BEEN  IDENTIFIED  AS  A PRIORITY?  - X 

C034601  SCHL0098  SCHOOL  OFFER  8TH  GR  STUDS  ALGEBRA  FOR  HS  CREDIT?  - X 

C037203  SCHL0099  SCHOOL  W/ SPECIAL  FOCUS  ON  ENGLISH?  - X 

C037306  SCHLOIOO  SCHOOL  FOLLOW  DISTRICT/STATE  ENGLISH  - X 

C039401  SCHL0101  SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  MATH?  - X 

C039402  SCHL0102  SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  SCIENCE?  - X 

C039403  SCHL0103  SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  READING?  - X 

C039404  SCHL0104  SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  ARTS?  - X 


Table  C-2  (continued) 

Summary  Table  of  Conditioning  Variable  Specifications 
for  the  1996  State  Assessment  in  Mathematics 


NAEP 

ID Description 

SCHL0105  SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  OTHER? 
SCHL0106  SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  NONE 
SCHL0107  8TH  GRADERS  IN  EXTRACURR  ACTS  FOR  MATH? 
SCHL0108  8TH  GRADERS  IN  EXTRACURR  ACTS  FOR  SCIENCE? 
SCHL0109  8TH  GRADERS  IN  EXTRACURR  ACTS  FOR  ENG/LANG 
SCHL01 10  8TH  GRADERS  IN  EXTRACURR  ACTS  FOR  ARTS? 

SCHLO 1 1 1 8TH  GRADERS  IN  EXTRACURR  ACTS  FOR  NONE  OF 
SCHLO 1 1 2 8TH  GRADERS  IN  SUMMER  PROGRAMS  IN  MATH? 

SCHLO  1 1 3 8TH  GRADERS  IN  SUMMER  PROGRAMS  IN  SCIENCE? 
SCHLO  114  8TH  GRADERS  IN  SUMMER  PROGRAMS  IN  ENG/LANG 
SCHLO  115  8TH  GRADERS  IN  SUMMER  PROGRAMS  IN  ARTS? 

SCHLO  116  8TH  GRADERS  IN  SUMMER  PROGRAMS  IN  NONE  OF 
SCHLO  1 1 7 WHAT  % OF  8TH  GRDS  HELD  BACK/REPEAT  8TH  GRADE? 
TCHRO 124  YRS  TOTAL  TAUGHT  MATH 

TCHRO 1 25  CURRICULUM  SPECIALIST  TO  HELP/ADVISE  IN  MATH? 
SCHLO  1 85  PERCENT  IN  NATIONAL  LUNCH  PROGRAM  FOR  GRADE 
SCHLO  1 86  PERCENT  IN  N ATION AL  LUNCH  PROGRAM  FOR  GRADE 


Grade  Grade 


REMRDGR  SCHLO  1 87  PERCENT  IN  REMEDIAL  READING  PRGRM  FOR  GRADE 

REMRDGR  SCHL01 88  PERCENT  IN  REMEDIAL  READING  PRGRM  FOR  GRADE 
REMMHGR  SCHLO  1 89  PERCENT  IN  REMEDIAL  MATH  PROGRAM  FOR  GRADE 

REMMHGR  SCHLO  190  PERCENT  IN  REMEDIAL  MATH  PROGRAM  FOR  GRADE 
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CONDITIONING  VARIABLE  ID:  BACK0001 


DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  OVERALL  (@ 


GRAND  MEAN 
N04,  S04 , N08 
OVERALL 
BKSER 
OTHER 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

GRAND  MEAN 


CONDITIONING  VARIABLE  ID:  BACK0002 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  MALE  (1 

002  FEMALE  (2 


DERIVED  SEX 

N04 , S04 , N08 , S08,  N12 

GENDER 

DSEX 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS : 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

MALE 

FEMALE 


CONDITIONING  VARIABLE  ID:  BACK0003 


DESCRIPTION : 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  WHI/AI/O  (1,5,6 

002  BLACK  (2 

003  HISPANIC  (3 

004  ASIAN  (4 


DERIVED  RACE/ ETHNICITY 

N04 , S04 , N08 , S08,  N12 

RACE/ETH 

DRACE 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  4 

NUMBER  OF  INDEPENDENT  CONTRASTS:  3 

RACE /ETHNICITY:  WHITE,  AMERICAN  INDIAN /ALASKAN  NAT’VE,  OTHER, 

MISSING,  UNCLASSIFIED 

RACE /ETHNICITY:  BLACK 

RACE /ETHNICITY:  HISPANIC 

RACE/ETHNICITY:  ASIAN 


CONDITIONING  VARIA3LE  ID:  BACK0004 

DESCRIPTION:  IF  HISPA 

GRADES/ ASSESSMENTS:  N04,  S04 

CONDITIONING  VAR  LABEL:  HISPANIC 

NAEP  ID:  . B003101 

TYPE  OF  CONTRAST:  CLASS 


001  NOT  HISP  (1 

002  MEXICAN  (2 

003  PUER  RIC  (3 

004  CUBN.OTH  (4,5 

005  HISP- ? (M 


IF  HISPANIC,  WHAT  IS  YOUR  HISPANIC  BACKGROUND? 

N04 , S04 , N08 , S08,  N12 
HISPANIC 

B003101  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


HISPANIC 

HISPANIC 

HISPANIC 

HISPANIC 

HISPANIC 


NOT  HISPANIC 

MEXICAN,  MEXICAN  AMERICAN,  CHICANO 
PUERTO  RICAN 
CUBAN,  OTHER 
MISSING 


CONDITIONING  VARIABLE  ID:  BACK0005 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


MSA/NON-MSA 

N04 , S04 , NO 8 , S08,  N12 

MSANAT 

TOL8 

CLASS 


001  MSA  (1.2, 3, 4, 7,9)  0 

002  NON  MSA  (5,6.8  ) 1 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


MSA 

NON -MSA 


CONDITIONING  VARIABLE  ID:  BACK0006 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  BIG  CTY5  (1 

002  MID  CTY5  (2.M 

003  FR/BTWN5  (3 

004  SML  TWN5  (4 

005  RURAL5  (5 


TYPE  OF  LOCALE  (5  CATEGORIES) 

N04,  S04 , N08 , S08,  N12 

TOL5 

TOL5 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


LARGE  CITY 
MID-SIZE  CITY 

URBAN  FRINGE  OF  LARGE  CITY 
SMALL  TOWN 

RURAL  (MSA  AND  NON-MSA) 


URBAN  FRINGE  OF  MID-SIZE  CITY 


CONDITIONING  VARIABLE  ID:  BACK0C07 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


DESCRIPTION  OF  COMMUNITY 
N04 , S04 , NOB,  S00,  N12 
DOC 
DOC 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

BIG  CITY 

(1 

) 000 

DOC: 

BIG  CITY 

002 

URBAN  FR 

(2 

) 100 

DCC: 

URBAN  FRINGE 

00  i 

MED  CITY 

( 5 , 9 , M 

) 010 

DOC: 

MEDIUM  CITY 

004 

SM  PLACE 

(4 

) 001 

DOC: 

SMALL  PLACE 

^ -3  A 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0008 

PARENTS*  HIGHEST  LEVEL  OF  EDUCATION 

N04 , SO  4 , NOB , S08 , N12 

PARED 

PARED  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

< HS 

(1 

) 0000 

002 

HS  GRAD 

(2 

) 1000 

003 

POST  HS 

(3 

) 0100 

004 

COL  GRAD 

(4 

) 0010 

005 

PARED-? 

{ 5 , M 

) 0001 

PARED:  LESS  THAN  HIGH  SCHOOL 

PARED:  HIGH  SCHOOL  GRADUATE 

PARED:  POST  HIGH  SCHOOL 

PARED : COLLEGE  GRADUATE 

PARED:  MISSING,  I DON'T  KNOW 


5 

4 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0010 
SCHOOL  TYPE  (PQ> 

N04 , S04,  NOB,  S08,  N12 

SCHTYPE 

SCHTYPE 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

NUMBER  OF  INDEPENDENT  CONTRASTS:  2 


001 

PUBLIC 

U 

) 00 

002 

PRIVATE 

( 2 . 4 , 5 , M 

) 10 

003 

CATHOLIC 

(3 

) 01 

SCHOOL  TYPE:  PUBLIC 

SCHOOL  TYPE:  PRIVATE,  BUREAU  OF  INDIAN  AFFAIRS,  DEPARTMENT  OF 

DEFENSE,  MISSING 
SCHOOL  TYPE:  CATHOLIC 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACKOOU 

INDIVIDUALIZED  EDUCATION  PLAN 

N04,  S04 , NOB.  S08,  N12 

IEP 

IEP 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  IEP • YES  (l  ) 0 

002  IEP -NO  (2  ) 1 


IEP:  YES 

IEP:  NO 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0012 

LIMITED  ENGLISH  PROFICIENCY 

N04,  S04 , NOB,  SOB,  N12 

LEP 

LEP 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  LEP-YES  (1  } O 

002  LEP-NO  (2  ) 1 


LEP:  YES 

LEP : NO 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACKOO 1 3 

CHAPTER  1 (BOOK  COVER) 

N04 , S04 , NOB,  SOB,  N12 

CHAPTER 1 

CKAP1 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  CHAP1-Y  (1  ) 0 

002  CHAP1-N  (2  ) 1 


CHAPTER  1 : YES 

CHAPTER  1:  NO 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACKOO l 4 

DO  YOU  RECEIVE  A FREE  OR 

N04,  S04,  NOB,  S08,  N12 

LUNCH 

SLUNCH 

CLASS 


REDUCED- PRICE  LUNCH? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  NOT  EL1G  (1  ) 0000 

002  RED  PRIC  (2  ) 1000 

003  FREE  (3  ) 0100 

004  INFO  N/A  ( 4 , M ) 0010 

005  SCH/REF  (5  ) 0001 


LUNCH  PROGRAM: 
LUNCH  PROGRAM: 
LUNCH  PROGRAM: 
LUNCH  PROGRAM: 
LUNCH  PROGRAM: 


NOT  ELIGIBLE 
REDUCED  PRICE 
FREE 

INFO  NOT  AVAILABLE 
SCHOOL  REFUAL 


5 

4 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACKOO 15 
HOW  OFTEN  DO 
N04,  S04 , NOB 
HOMELANG 
B003201 
CLASS 


THE  PEOPLE  IN  YOUR  HOME  SPEAK  A LANGUAGE  OTHER  THAN  ENGLISH? 
, SOB,  N12 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


4 

3 


001 

HI.  NEVER 

( 1 

) ooo 

002 

HL-SOME 

(2 

) 100 

003 

HL-ALWAY 

( 3 

) 010 

004 

HL-'> 

(M 

) 001 

HOMELANG : 
HOMELANG : 
HOMELANG : 
HOMELANG : 


NEVER 

SOMETIMES 

ALWAYS 

MISSING 


best  copy  AVAILABLE 
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331 


CONDITIONING  VARIABLE  ID; 
DESCRIPTION: 
GRADES/ASSESSMENTS ; 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0016 

DO  YOU  HAVE  YOUR  OWN  STUDY  DESK  OR  TABLE  AT  HOME? 

N04,  S04 , N08 , S08 , N12 

B008901  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

B008901Y 

(01 

) 000 

002 

B00890 IN 

(02 

) 100 

003 

B008901I 

(03 

) 010 

004 

B008901M 

<M 

) 001 

YES 

NO 

IDK25  (FILL  1 SHELF) 
MISSING 


4 

3 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0017 

HOW  MUCH  TELEVISION  DO  YOU  USUALLY  WATCH  EACH  DAY?  (LINEAR) 
N04 , S04 , N08 , S08 , N12 
TVWATCHL 
B001801 
LINEAR 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


7 

1 


001 

TVLIN-G 

(1 

) 0 

002 

TVLIN-1 

(2 

) 1 

0C3 

TVLIN-2 

(3 

) 2 

004 

TVLIN-3 

{ 4 , M 

) 3 

005 

TVLIN-4 

(5 

) 4 

006 

TVLIN-5 

(6 

) 5 

007 

TVLIN-6 

(7 

) 6 

TV  WATCHING  (LINEAR) 
TV  WATCHING  (LINEAR) 
TV  WATCHING  (LINEAR) 
TV  WATCHING  (LINEAR) 
TV  WATCHING  (LINEAR) 
TV  WATCHING  (LINEAR) 
TV  WATCHING  (LINEAR) 


(0  TO  6+  HOURS  PER  DAY) 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0018 

HOW  MUCH  TELEVISION  DO  YOU  USUALLY  WATCH  EACH  DAY?  (QUADRATIC) 

N04 , S04 , N08 , S08,  N12 
TVWATCHQ 

B001801  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

QUADRATIC  NUMBER  OF  INDEPENDENT  CONTRASTS: 


1 

1 


COI  TV-QUAD  { 1-7 , M=4 


1.0  + -2 . 0 *X  ♦ 1 . 0 *X* * 2 


TV  WATCHING  (QUADRATIC) 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0019 

HOMEWORK  ASSIGNED?: 

N04 , S04 , NOB,  SOB, 

HWASSIGN 

B006601 

CLASS 


BASED  ON  TIME  SPENT  ON  HOMEWORK  EACH  DAY. 
N12 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

HW-MISS 

(M 

) 00 

HOMEWORK 

ASSIGNED?: 

MISSING 

002 

HW-NO 

(1 

) 10 

HOMEWORK 

ASSIGNED? : 

NO 

0C  3 

HW- YES 

(2-5 

) 01 

HOMEWORK 

ASSIGNED? : 

YES 

3 

2 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES ■ ASSESSMENTS : 
CONDITIONING  VAP  LABEL: 
NAEP  ID: 

TYPE  C-F  CONTRAST: 


BACK0020 

HOW  MUCH  TIME  DO  YOU  USUALLY  SPEND  ON  HOMEWORK  EACH  DAY?  (LINEAR) 
N04 , S04,  N08 , S08,  N12 
HOMEWRKL 
B006601 
LINEAR 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


4 

1 


00 1 HWLIN-0  (1  2.M  ) 0 

002  HWLIN-1  (3  ) 1 

003  HWL1N-2  (4  ) 2 

004  HWLIN-3  (5  ) 3 


HOMEWORK  (LINEAR) 
HOMEWORK  (LINEAR) 
HOMEWORK  (LINEAR) 
HOMEWORK  (LINEAR) 


DON’T  HAVE  ANY,  DON’T  DO  ANY , MISSING 
1/2  HOUR  OR  LESS 
1 HOUR 

MORE  THAN  1 HOUR 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK002 1 

HOW  MUCH  TIME  DO  YOU  USUALLY  SPEND  ON  HOMEWORK  EACH  DAY  (QUADRATIC) 


N04 , S04 , NO 8 , SOB,  N12 

HOMEWRKQ 

B006601 

SCALE 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


4 

1 


001  HWQUAD-0  ( 1 , 2 , M ) 0 

002  HWQUAD-1  (3  ) 1 

003  HWQUAD-2  (4  ) 4 

004  HWQIJAD-3  (5  ) 9 


HOMEWORK  (QUADRATIC) 
HOMEWORK  (QUADRATIC) 
HOMEWORK  (QUADRATIC) 
HOMEWORK  (QUADRATIC) 


DON’T  HAVE  ANY,  DON’T  DO  ANY,  MISSING 
1/2  HOUR  OR  LESS 
1 HOUR 

MORE  THAN  1 HOUR 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

1'YIL  * K CONTRAST- 


BACK0022 

NUMBER  OF  ITEMS  IN  THE  HOME  (NEWSPAPER,  > 25  BOOKS,  ENCYCLOPEDIA,  MAGAZINES)  (DERIVED) 

N04,  S04 , NOB,  SOB,  N12 

HOMEITMS 

HOMEEN2  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS:  2 


00J  H1TEM'  "2  ( 1 , M 1 00 


ITEMS  IN  HOME:  ZERO  TO  TWO  ITEMS,  MISSING 


294 


o nt  r) 

U k"  f_0 


002  HITEM=  3 (2 

003  HITEM-4  (3 


) 10 
) 01 


ITEMS  IN  HOME:  THREE  ITEMS 

ITEMS  IN  HOME:  FOUR  ITEMS 


CONDITIONING  VARIABLE  ID:  BACK0023 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


DOES  MOTHER  OR  STEPMOTHER  LIVE  AT  HOME  WITH  YOU? 

N04 , S04 , N08 , SG8.  N12 
MOM9HOME 

B00560 1 TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  MOMHOM-Y  U 

002  MOMHOM-N  (2 

003  MOMHOM-?  <M 


MOTHER  AT  HOME 
MOTHER  AT  HOME 
MOTHER  AT  HOME 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


DOES  FATHER  OR  STEPFATHER  LIVE  AT  HOME  WITH  YOU’ 

N04,  S04,  NOB , S0R,  N12 
DAD9HOME 

B005701  TOTAL  NUMBER  OF  SPECIFIED  CONTRAST! 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  DADHOM-Y  (1 

002  DADHOM-N  (2 

003  DADHOM-?  (M 


FATHER  AT  HOME 
FATHER  AT  HOME 
FATHER  AT  HOME 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  BACK0025 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  MANY  DAYS  OF  SCHOOL  MISSED  LAST  MONTH? 

N04,  S04.  N08 , SOB,  N12 
SCH  MISS 

S004001  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  MISS- >2  < 3 . 4. 5.M 

002  MISS  - 2<  (1,2 


DAYS  OF  SCHOOL  MISSED:  3-4.  5-10,  10  OR  MORE  DAYS.  MISSli 

DAYS  OF  SCHOOL  MISSED:  0-1.  2 DAYS 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LA3EL; 
NAEP  ID. 

TYPE  OF  CONTRAST: 


HOW  LONG  LIVED  IN  THE  UNITED  STATES’ 

NO 4 , S04 , NOB,  S08,  N12 
YRSINUSA 

B008001  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


C01  ALL  MY  L (1 

002  USA  >5  (2 

003  USA  3b  (3 

004  USA  'i  (4 

00b  USA- ’ (M 


) 0000 
) 1000 
) 0100 
) 0010 
> 0001 


ALL  MY  LIFE 

LIVED  IN  US  MORE  THAN  5 YEARS 
LIVED  IN  US  3-5  YEARS 
LIVED  IN  US  LESS  THAN  3 YEARS 
LIVED  IN  US  MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL' 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  MANY  GRADES  IN  THIS  STATE’  t 4TH  GRADE ) 

N04 , S04 
STGRADE4 

B007601  TOTAL  NUMBER  OF  SPECIFIED  CONTRAST: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  STCRD< 1 

002  STGRD1 - 

003  STGRD3  > 


GRADES  IN  STATE 
GRADES  IN  STATE 
GRADES  IN  STATE 


LESS  THAN  1 GRADE,  MISS  IN'. 
1-2  GRADES 
3 OR  MORE  GRADES 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION . 

GRADES , ASSESSMENTS : 
CONDITIONING  VAR  LABEL. 

HA HP  ID: 

TYPE  OF  CONTRAST: 


BACK0C29 

HOW  MANY  TIMES  HAVE  YOU  CHANGED  SCHOOLS  IN  PAST  TWO  YEARS  BECAUSE  YOU  MOVED 
N04 , S04,  NOB,  SCR,  N12 
SCH  CHGS 

BOO?  301  TOTAL  NUMBER  OF  SPECIFIED  CONTRAST!;  4 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS:  J 


001  SCHCHG- 0 II 

002  SCHCHG-1  (2 

003  SCHCHG-2  (3 

004  SCHCHG- 3 (4.M 


SCHOOL  CHANGES: 
SCHOOL  CHANGES 
SCHOOL  CHANGES 
SCHOOL  CHANGES 


NONE 

ONE 

TWO 

THREE  OR  MORE.  MISSIN 


CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  he  CONTRAST 

001  DIS0HOM1  (1 
00?  PISRHOM2  (2 


BACK0030 

HOW  OFTEN  DO  YOU  DISCUSS  THINGS  STUDIED  IN  SCHOOL  WITH  SOMEONE  AT  HOME’ 

N04,  504 , N08 . SOP.  N12 

DISC9HOM 

0007 401  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

NUMBER  OF  INDEPENDENT  CONTRASTS  • 


DISCUSS  STUDIES  AT  HOME: 
DISCUSS  STUDIES  AT  HOME 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 


9 o 


003  DIS0HOM3  (3 

004  DIS0HOM4  (4,M 


) 010 
) 001 


DISCUSS  STUDIES  AT  HOME:  ONCE  OR  TWICE  A MONTH 
DISCUSS  STUDIES  AT  HOME:  NEVER  OR  HARDLY  EVER,  MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK003 1 

ABOUT  HOW  MANY  PAGES  A DAY  DO  YOU  HAVE  TO  READ  FOR  SCHOOL  AND  HOMEWORK? 

N04,  S04 , NOS , S08,  N12 

PGSREAD1 

B001101  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  PGS<6 , ? ( 5 , M ) 0 

002  PGS>5  ( i , 2 . 3 , 4 ) 1 


PAGES  READ:  5 OR  FEWER  A DAY,  MISSING 

PAGES  READ:  6-10,  11-15,  16-20,  20  OR  MORE 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0032 
ABOUT  HOW 
N04,  S04 , 
PGSREAD2 
B001101 
CLASS 


MANY  PAGES  A DAY  DO  YOU  HAVE  TO  READ  FOR  SCHOOL  AND  HOMEWORK? 
N08 , S08 . N12 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  PG3<  1 1 , ■>  (4. 5.M  ) 0 

002  PGS>10  (1,2,3  ) 1 


PAGES  READ:  6-10,  5 OR  FEWER  A DAY,  MISSING 

PAGES  READ:  11-15,  16-20,  20  OR  MORE 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK004  8 

INTERACTION:  GENDER  BY  RACE/ ETHNICITY 

N04 , S04 , N08,  S08,  N12 

GEND/RAC 

N/A  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


8 

3 


001 

G/R 

11 

(11 

) 010101 

GEND/RAC 

INTACT : 

1. 

MALE 

1. 

WHI/AI/O 

002 

G/R 

12 

(12 

} -10000 

GEND/RAC 

INTACT: 

1. 

MALE 

2. 

BLACK 

003 

G/R 

13 

(13 

i 00-100 

GEND/RAC 

INTACT : 

1 . 

MALE 

3. 

HISPANIC 

004 

G/R 

14 

(14 

) 0000-1 

GEND/RAC 

INTACT : 

1. 

MALE 

4, 

ASIAN 

005 

G/R 

21 

<21 

) -1-1-1 

GEND/RAC 

INTACT : 

2. 

FEMALE 

1. 

WHI/AI/O 

006 

G/R 

22 

(22 

) 010000 

GEND/RAC 

INTACT: 

2 . 

FEMALE 

2. 

BLACK 

007 

G/R 

23 

(23 

) 000100 

GEND/RAC 

INTACT : 

2. 

FEMALE 

3. 

HISPANIC 

008 

G/R 

24 

(24 

) 000001 

GEND/RAC 

INTACT : 

2. 

FEMALE 

4. 

ASIAN 

10 

4 


001 

G/T 

11 

(11 

) 01010101 

GEND/TOL 

INTACT: 

1. 

MALE 

1 . 

BIG  CTY5 

002 

G/T 

12 

(12 

i -1000000 

GEND/TOL 

INTACT : 

1 . 

MALE 

2 . 

MID  CTY5 

003 

G/T 

12 

(13 

) 00-10000 

GEND/TOL 

INTACT : 

1. 

MALE 

3. 

FR/BTWN5 

004 

G'T 

14 

(14 

) 0000-100 

GEND/TOL 

INTACT: 

1. 

MALE 

4. 

SML  TWN5 

005 

G/T 

1 r' 

(15 

I 000000-1 

GEND/TOL 

INTACT : 

1 . 

MALE 

5. 

RURAL 5 

006 

G-  T 

21 

(21 

) -l-l-l-l 

GEND/TOL 

INTACT : 

2, 

FEMALE 

1 . 

BIG  CTY5 

007 

G/T 

22 

(22 

) 01000000 

GEND/TOL 

INTACT: 

2. 

FEMALE 

2. 

MID  CTY5 

008 

G.  T 

23 

(23 

) 00010000 

GEND/TOL 

INTACT : 

2. 

FEMALE 

3. 

FR/BTWN5 

009 

G-T 

24 

(24 

) 00000100 

GEND/TOL 

INTACT: 

2 . 

FEMALE 

4 

SML  TWN5 

010 

G,  T 

25 

(25 

) 00000001 

GEND/TOL 

INTACT: 

2. 

FEMALE 

5. 

RURAL 5 

CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK 00 4 9 

INTERACTION:  GENDER  BY  TYPE  OF  LOCALE  (5  CATEGORIES) 

N04,  S04 , NOS . S08 , N12 
GEND/TOL 

N/A  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0050 

INTERACTION:  GENDER  BY  PARENTS'  EDUCATION 

N04 , S04 , NOS,  SOB , N12 

GEND/PAR 

N/A  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


10 

4 


001 

G P 

I 1 

( 1 1 

002 

C:  P 

12 

(12 

003 

G;  P 

13 

(13 

C04 

G,  P 

14 

(14 

005 

G,  P 

15 

(15 

006 

G,  P 

21 

(21 

007 

G.  P 

22 

(22 

008 

0/  P 

2 3 

(23 

009 

G P 

24 

(24 

010 

G • P 

25 

( 2 5 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

-SPADES /ASSESSMENTS : 
CONDITIONING  VAR  LAPEL: 
NAEP  ID: 

TYPE  OK  CONTRAST. 


) 01010101 
) -1000000 
» 00-10000 
) 0000-100 
) 000000-1 
) -l-l-l-l 
) 01000000 
) 00010000 
) 00000100 
) 00000001 


BACK005 1 

INTERACTION:  GENDER  BY 

N04 , S04 , N0H,  GrtP . N1? 
U END; SOU 
N/A 

INTERACTION 


SCHOOL 


GEND/PAR 

GEND/PAR 

GEND/PAR 

GEND/PAR 

GEND/PAR 

GEND/PAR 

GEND/PAR 

GEND/PAR 

GEND/PAR 

GEND/PAR 


INTACT : 

1 . 

MALE 

1 . 

< HS 

INTACT: 

1 . 

MALE 

2 . 

HS  GRAD 

INTACT: 

1 . 

MALE 

3 . 

POST  HS 

INTACT: 

1 . 

MALE 

4 . 

COL  GRAD 

INTACT: 

1 . 

MALE 

5. 

PARED-  ? 

INTACT: 

2 . 

FEMALE 

1. 

< HS 

INTACT: 

2 . 

FEMALE 

2. 

HS  GRAD 

INTACT: 

2. 

FEMALE 

3. 

POST  HS 

INTACT: 

2 . 

FEMALE 

4. 

COL  GRAD 

INTACT: 

2 . 

FEMALE 

5. 

PARED-? 

TYPE 


TOTAL  HUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


6 

2 


2% 


BEST  COPY  AVAILABLE 


O n 4 

Gy/  7 


001 

G/S 

11 

(11 

) 0101 

GEND/SCH 

INTACT : 

1 . 

MALE 

1 . 

PUBLIC 

002 

G/S 

12 

(12 

) -100 

GEND/SCH 

INTACT : 

1. 

MALE 

2 . 

PRIVATE 

003 

G/S 

13 

(13 

) 00-1 

GEND/SCH 

INTACT : 

1 . 

MALE 

3 . 

CATHOLIC 

004 

G/S 

21 

<21 

) -1-1 

GEND/SCH 

INTACT : 

2 . 

FEMALE 

1 . 

PUBLIC 

005 

G/S 

22 

(22 

) 0100 

GEND/SCH 

INTACT : 

2. 

FEMALE 

2. 

PRIVATE 

006 

G/S 

23 

(23 

) 0001 

GEND/SCH 

INTACT : 

2 . 

FEMALE 

3. 

CATHOLIC 

CONDITIONING  VARIABLE  ID:  BACK0052 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


INTERACTION:  RACE/ ETHNICITY  BY  TYPE  OF  LOCALE  (5  CATEGORIES f 

N04,  S04 , NOR,  S08,  N12 

RACE/TOL 

N/A  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


R/T  11 
R/T  12 
R/T  13 
R/T  14 
R/T  15 
R/T  21 
R/T  22 
R/T  23 
R/T  24 
R/T  25 
R/T  31 
R/T  32 
R/T  33 
R/T  34 
R/T  35 
R/T  41 
R/T  42 
R/T  43 
R/T  44 
R/T  45 


) 010101010101010101010101 
) -1000000-1000000-1000000 
) 00-1000000-1000000-10000 
) 0000-1000000-1000000-100 
) 000000-1000000-1000000-1 
) -1-1-1-10000000000000000 
) 010000000000000000000000 
) 000100000000000000000000 
) 000001000000000000000000 
) oooooooioooooooooooooooo 
) 00000000-1-1-1-100000000 
) oooooooooioooooooooooooo 
) 000000000001000000000000 
) 000000000000010000000000 
) 000000000000000100000000 
) OOOCOOOOOOOOOCOO-l-l-l-l 
) 000000000000000001000000 
) 000000000000000000010000 
) 000000000000000000000100 
) 000000000000000000000001 


RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 

RACE/TOL 


INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 


1 . 

WHI/AI/O 

1 . 

BIG  CTY5 

1 . 

WHI/AI/O 

2 . 

MID  CTY5 

1. 

WHI/AI/O 

3 . 

FR/BTWNS 

1. 

WHI/AI/O 

4 . 

SML  TWN5 

1 . 

WHI/AI/O 

5. 

RURAL5 

2. 

BLACK 

1. 

BIG  CTY5 

2. 

BLACK 

2 . 

MID  CTY5 

2, 

BLACK 

3 . 

FR/BTWN5 

2. 

BLACK 

4 . 

SML  TWN5 

2. 

BLACK 

5. 

RURAL 5 

3. 

HISPANIC 

1 . 

BIG  CTY5 

3. 

HISPANIC 

2 . 

MID  CTY5 

3. 

HISPANIC 

3. 

FR/BTWN5 

3. 

HISPANIC 

4 . 

SML  TWN5 

3. 

HISPANIC 

5. 

RURAL5 

4 . 

ASIAN 

1 . 

BIG  CTY5 

4. 

ASIAN 

2. 

MID  CTY5 

4 . 

ASIAN 

3 . 

FR/BTWN5 

4. 

ASIAN 

4 . 

SML  TWN5 

4. 

ASIAN 

5 _ 

RURAL5 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0053 

INTERACTION:  RACE /ETHNICITY  BY  PARENTS*  EDUCATION 

N04 , S04,  NOS,  S08 , N12 
RACE /PAR 

M/A  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


R/P  11 
R/P  12 
R/P  13 
R/P  14 
R/P  15 
R/P  21 
R/P  22 
R/P  23 
R/P  24 
R/P  25 
R/P  31 
R/P  32 
R/P  33 
R/P  34 
R/P  35 
R/P  41 
R/P  42 
R/P  43 
R/P  44 
R/P  45 


) 010101010101010101010101 
) -1000000-1000000-1000000 
) 00-1000000-1000000-10000 
) 0000-1000000-1000000-100 
) 000000-1000000-1000000-1 
) -1-1-1-10000000000000000 
) 010000000000000000000000 
) 000100000000000000000000 
) 000001000000000000000000 
) OOOOOOOIOOOOOOOOOOOOOOOO 
) 00000000-1-1-1-100000000 
) oooooooooioooooooooooooo 
) oooooooooooioooooooooooo 
) oooooooooooooioooooooooo 
) 000000000000000100000000 
) 0000000000000000-1-1-1-1 
) 000000000000000001000000 
] oooooooooooooooooooioooo 
) oooooooooooooooooooooioo 
) 000000000000000000000001 


RACE/ PAR 
RACE/ PAR 
RACE/ PAR 
RACE /PAR 
RACE/ PAR 
RACE/ PAR 
RACE/ PAR 
RACE/ PAR 
RACE/ PAR 
RACE /PAR 
RACE /PAR 
RACE/ PAR 
RACE /PAR 
RACE/ PAR 
RACE/ PAR 
RACE/ PAR 
RACE /PAR 
RACE /PAR 
RACE/ PAR 
RACE ' PAR 


INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 


1. 

WHI/AI/C 

1. 

< HS 

1 . 

WHI  'AI/O 

2. 

HS  GRAD 

1 . 

WHI/AI  O 

3. 

POST  HS 

1 . 

WHI/AI/O 

4 

COL  GRAD 

1 . 

WHI  'AI.'O 

5. 

PARED- ? 

2. 

BLACK 

1 . 

< HS 

2. 

BLACK 

2 . 

HS  GRAD 

2 . 

BLACK 

3 . 

POST  HS 

2 

BLACK 

4. 

COL  GRAD 

2. 

BLACK 

5. 

PARED  - ■> 

3. 

HISPANIC 

1 . 

< HS 

3. 

HISPANIC 

2. 

HS  GRAD 

3. 

HISPANIC 

3. 

POST  HS 

3. 

HISPANIC 

4 . 

COL  GRAD 

3. 

HISPANIC 

5. 

PARED-? 

4. 

ASIAN 

1 . 

< HS 

4 . 

ASIAN 

-> 

HS  GRAD 

4 . 

ASIAN 

3 . 

POST  HS 

4. 

ASIAN 

4 . 

COL  GRAD 

4 . 

ASIAN 

5. 

PARED-’ 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0054 

INTERACTION:  RACE; ETHNICITY  BY  SCHOOL  TYPE 

N04,  S04 , NOB,  SOB,  N12 

RACE/SCH 

M/a  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  R/S  11 

002  R/S  12 

003  R/S  13 

004  R/S  21 

005  R/S  22 

006  R/S  23 
00**  R/S  31 

008  R/S  32 

009  R/S  33 

010  P/S  41 

011  R/S  42 

012  R/S  43 


) 010101010101 
) -100-100-100 
) 00-100-100-1 
) -1-100000000 
) 010000000000 
) 000100000000 
) 0000-1-10000 
) oooooioooooo 
) 000000010000 
) 00000000  1 1 
] 000000000100 
) 000000000001 


RACE/SCH 

INTACT: 

1 . 

WHI/AI/O 

1 . 

PUBLIC 

RACE/ SCH 

INTACT: 

1 . 

WHI/AI/O 

2. 

PRIVATE 

RACE/SCH 

INTACT: 

1 . 

WHI/AI/O 

3 . 

CATHOLIC 

RACE/SCH 

INTACT: 

2. 

BLACK 

1 . 

PUBLIC 

RACE/SCH 

INTACT : 

2. 

BLACK 

2 . 

PRIVATE 

RACE/SCH 

INTACT : 

2. 

BLACK 

3 . 

CATHOLIC 

RACE/SCH 

INTACT: 

3 . 

HISPANIC 

1 . 

PUBLIC 

RACE/SCH 

I NT ACT : 

3 . 

HISPANIC 

2 . 

PRIVATE 

RACE/SCH 

INTACT: 

3 . 

HISPANIC 

3. 

CATHOLIC 

RACE/SCH 

INTACT : 

4 . 

ASIAN 

1 

PUBLIC 

RACE  < SCH 

INTACT: 

4 . 

ASIAN 

"> 

PRIVATE 

RACE ‘SCH 

INTACT. 

4 . 

ASIAN 

CATHOLIC 

c-  o rt 
O <j>  o 


BEST  COPY  AVAILABLE 


291 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION : 

GRADES / ASSESSMENT S : 
CONDITIONING  VAR  LAEEL : 

;:aep  ID: 

TYPE  Or  CONTRAST: 


BACK005S 

INTERACTION:  TYPE  OF  LOCALE  (5  CATEGORIES)  BY  PARENT'S  EDUCATION 

N04.  S04 , NOB,  SOB,  NI2 
TOL5  / °AR 

N/A  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


> 010 10101 010101010 101010 10 1010101 
) -1000000-1000000-1000000-1000000 
) 00-1000000-1000000-1000000-10000 
» 0C00- 1000000- 1000000-1000000-100 
I 0G0000- 1000000- 1000000-1000000-1 
) -1-1-1-1000000000000000000000000 
) 01000000000000000000000000000000 
> 00010000000000000000000000000000 
1 C0000100000000000000000000000000 
) 00000001000000000000000000000000 
) 00000000-1-1-1-10000000000000000 
) C0000000010000000000000000000000 

) oooooooooooioooooooooooooooooooo 
) ooocoooooooooioooooooooooooooooo 

i G0000000000000010000000000000000 
) 0000000000000000-1-1-1-100000000 
i 00000000000000000100000000000000 
) 00000000000000000001000000000000 
> 00000000000000000000010000000000 
1 00000000000000000000000100000000 
) OOOOOGOOOOOOOOOOOCOOOOOO-l- 1-1-1 
> 00000000000000000000000001000000 
) 00000000000000000000000000010000 
) COCOOOOOOOOOOOOOOOOOOOOOOOOOOIOO 
» C-OOOOOOOCCOOOOOOOOOOOOOOOOOOOOOl 


TOL5/PAR 
TOL5/PAR 
TOL5/PAR 
TOLS/PAR 
TOL5/PAR 
TOL5/PAR 
TOL5/PAR 
TOLS/PAR 
TOL5/PAR 
TOL5/PAR 
TOL5/PAR 
TOL5/PAR 
TOL5/PAR 
TOL5/PAR 
TOL5/PAR 
TOLS/PAR 
TOL5/PAR 
TOLS/PAR 
TOL5 / PAR 
TOLS/PAR 
TOLS/PAR 
TOLS/PAR 
TOL5 / PAR 
TOL5 / PAR 
TOL5  /'  PAR 


INTACT:  1 
INTACT:  1 
INTACT:  1 
INTACT:  1 


INTACT 

INTACT 

INTACT 

INTACT 

INTACT 

INTACT 


INTACT:  4 
INTACT:  4 
INTACT : 4 
INTACT:  4 


INTACT:  5 
INTACT:  5 


BIG  CTYS  1. 
BIG  CTY5  2. 
BIG  CTY5  3. 
BIG  CTY5  4. 


< HS 
HS  GRAD 
POST  HS 
COL  GRAD 


y:n:it:cn:ng  variable  id:  backooss 

DE.rjF. : ft:  :n  . interact 

3 PAD EG  ASSESSMENTS.  N04 , S04 

C.-NDIT;  .-KING  YAK  LABEL:  TOL5/SCH 


1. 

BIG  CTYS 

5. 

PARED-? 

2 . 

MID  CTYS 

1 . 

< HS 

2 . 

MID  CTYS 

2. 

HS  GRAD 

2. 

MID  CTYS 

3 . 

POST  HS 

2. 

MID  CTY5 

4. 

COL  GRAD 

2. 

MID  CTY5 

5. 

PARED-? 

3 . 

FR/BTWN5 

1. 

< HS 

3 . 

FR/BTWN5 

2. 

HS  GRAD 

3 . 

FR/BTWN5 

3. 

POST  HS 

3. 

FR/BTWN5 

4. 

COL  GRAD 

3. 

FR/BTWN5 

5. 

PARED-? 

4 . 

SML  TWN5 

1 . 

< HS 

4 . 

SML  TWN5 

2 . 

HS  GRAD 

4. 

SML  TWN5 

3 

POST  HS 

4 . 

SML  TWNS 

4. 

COL  GRAD 

4 . 

sml  twns 

5. 

PARED-  “> 

5. 

RURALS 

1. 

< HS 

5. 

RURAL5 

2. 

HS  GRAD 

5. 

RURALS 

3. 

POST  HS 

5. 

RURAL 5 

4. 

COL  GRAD 

5. 

P.URALS 

S. 

PARED- 7 

HOCL  TYPE 

NO 4 , S04 , NOS , SOS,  N12 
TOL5 / SCH 


INTERACTION 


TOTAL  NUMBER  Or  SPECIFIED  CONTRASTS : 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


i 7 11 

t 1 1 

> oicaoioioioioioi 

TCL5/SCH 

INTACT: 

1 . 

BIG  CTYS 

1 . 

PUBLIC 

T f. 

■ ir 

o 

o 

o 

o 

o 

o 

o 

o 

TOL5 / SCH 

INTACT: 

1 . 

BIG  CTYS 

2. 

PRIVATE 

} 7 f.  : ; 

} i 

» CO-1QO-1O0-1OO-1 

TOL5 / SCH 

INTACT: 

1 . 

BIG  CTYS 

3. 

CATHOLIC 

4 I '■  2 1 

•21 

» -1-1000000000000 

TOL5/SCH 

INTACT: 

2 . 

MID  CTY5 

1 . 

PUBLIC 

^ T ; 

• ..1 

1 OlOCOOOOOOQOOOOO 

TOL5/SCH 

INTACT: 

2 . 

MID  CTYS 

2 . 

PRIVATE 

t - ;*  • 

: _ * 

1 CCOIOOCOOOOOOOGO 

TOLS/SCH 

INTACT: 

2 . 

MID  CTYS 

3. 

CATHOLIC 

7 •:  * : 

• l # 

> OOCO-l-iOOOOOOOO 

TOLS/SCH 

INTACT: 

3 . 

FR/BTWN5 

1 . 

PUBLIC 

- ” ••  : ; 

, : ^ 

t 0000010000000000 

TOL5/SCH 

INTACT : 

3. 

FR/BTWN5 

2. 

PRIVATE 

- 7‘  . • • 

< i - 

> 0000000100000000 

TOL5/SCH 

INTACT: 

3 . 

FR/BTWN5 

3. 

CATHOLIC 

I : ’ 4 1 

< i » 

« 10000000-1-10000 

TOL5/SCH 

INTACT : 

4 . 

SML  TWNS 

1 . 

PUBLIC 

„ T * 4.. 

< 4, 

» G00G00000100G0C0 

TOLS/SCH 

INTACT: 

4 . 

SML  TWN5 

2. 

PRIVATE 

< 4 : 

) COOOOOOOOOOIOOCO 

TOL5/SCH 

INTACT : 

4 . 

SML  TWNS 

3. 

CATHOLIC 

} 7 . 

< 5 : 

i «: ooooooooooo-i-i 

TOLS/SCH 

INTACT : 

5. 

RURALS 

1 . 

PUBLIC 

1 T S S.. 

t t ~ 

) coooooooocoooico 

TOLS/SCH 

INTACT : 

5. 

RURALS 

2 . 

PRIVATE 

S T S ? 

. =.  3 

> cr'coooocoooooooi 

TOLS / SCH 

INTACT : 

5. 

RURALS 

3. 

CATHOLIC 

■ :.t  • t : •: : no  vap  : ar  l v. 
ES "F I P7 1 IN ■ 

a:;  'fcsment - • 


EACKDOS? 

I NTERACTI ON  : PARENTS  * El 

NO 4.  SO 4,  NCR  SCR,  N12 
PARE. SCH 
N A 

interact; ?n 


icatjcn  by  school  type 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS : 


: i 

..  1 1 

. ■ ■ 

.•  C1C1C1C101C10101 

PARE /SCH 

INTACT 

1 . 

< HS 

1 , 

PUBLIC 

_ ; 

« . ; 

i -:go-icd- 100-100 

PARE/ SCH 

INTACT 

1 . 

*•  HS 

2 . 

PRIVATE 

c ; * 

i > 

) C0- 100-100-100-1 

PARE, SCH 

INTACT 

1 . 

< HS 

3 

CATHOLIC 

•i 

' : i ^ 

, : 

i -1-1000000000000 

PARE7  SCH 

INTACT 

2. 

HS  GRAD 

1 . 

PUBLIC 

1 

( ; . 

> 01 00000000000000 

PARE /SCH 

INTACT 

2 . 

HS  GRAD 

2 . 

PRIVATE 

> 

r . t 

r ; ! 

- 0001000000000000 

PARE/ SCH 

INTACT 

2 . 

HS  GRAD 

3 . 

CATHOLIC 

■ I 

* . j : 

< • ; 

» 0000-1-100000000 

PARE/ SCH 

INTACT 

3 . 

POST  HS 

1 . 

PUBLIC 

- i 

' " 

■ '•  £ 

> 0000010000000000 

PARE /SCH 

INTACT 

3 . 

POST  HS 

2 . 

PRIVATE 

. » . 

i * • 

» C000000100000000 

PARE/ SCH 

INTACT 

3 . 

POST  HS 

3 . 

CATHOLIC 

1 

• 4 : 

■■  4 : 

» 00000000-1-10000 

PARE/ SCH 

INTACT 

4 . 

COL  GRAD 

1 . 

PUBLIC 

1 1 

.j , 

• 4. 

t COOOOODOQIOOOOOO 

PARE/ SCH 

INTACT 

4 . 

COL  GRAD 

2 . 

PRIVATE 

. 

’ : 4 J 

'4  ' 

• jCOOOOOOOOOIOOOO 

PAPE /SCH 

INTACT 

4 . 

COL  GRAD 

3 . 

CATHOLIC 

. f ’ 

> 000000000000 -1-1 

PARE,  SCH 

INTACT 

S. 

PARED-? 

1 . 

PUBLIC 

4 

.:  \ . 

) 0000000000000100 

PARE' SCH 

INTACT 

S . 

PARED-? 

2 . 

PRIVATE 

> 

i 1 ' 

t ’000000000000001 

PARE  SCH 

INTACT 

S . 

PARED-  •> 

3 . 

CATHOLIC 

-;ir: 

./a 

I'ACKO-'*  0 

I NT  P FACT  I *N  TI.TfP  P’ 

' MATH 

■vf't::  tap  u 

; 3 THIS 

YFAf 

A;  I-..'  A.  VK'i.’MEN  I*  * . 

n:*:t;  n:n-  vap  lap. 


BEST  COPY  AVAILABLE 


NAEP  ID: 
TYPE  OF 

CONTRAST : 

N/A 

INTERACTION 

001 

G/ 

11 

(11 

) 01010101010101 

002 

G/ 

12 

(12 

) -1000000000000 

003 

G f 

13 

1 13 

) 00-10000000000 

004 

G/ 

14 

(14 

) 0000-100000000 

005 

G/ 

15 

{15 

) 000000-1000000 

006 

G; 

16 

(16 

) 00000000-10000 

007 

G/ 

17 

(17 

) 0000000000-100 

00R 

G/ 

1R 

(IR 

) 000000000000-1 

009 

G/ 

21 

(21 

> -l-l-l-l-l-l-l 

010 

G/ 

22 

(22 

) 01000000000000 

Oil 

G/ 

23 

(23 

) 00010000000000 

012 

G. 

24 

(24 

) 00000100000000 

013 

0/ 

25 

(25 

) 00000001000000 

014 

G / 

26 

(26 

) 00000000010000 

015 

G/ 

27 

(27 

) 00000000000100 

016 

G / 

2 R 

( 2 P 

) 00000000000001 

CONDITIONING 

VARIABLE  ID: 

BACK0060 

DESCRIPTION: 

INTERACTION:  RACE/ETHNI 

GRADES /ASSESSMENTS : 

NOR,  SOR 

CONDITIONING 

VAF.  LABEL: 

RACE/ 

NAEP  ID: 

N/A 

TYPE  OF 

CONTRAST : 

INTERACTION 

001 

R- 

11 

(11  ] 

333333333333333333333 

002 

R/ 

12 

(12  ] 

122222212222221222222 

003 

R / 

13 

(13  ] 

212222221222222122222 

004 

R 

14 

(14  ] 

221222222122222212222 

005 

R. 

15 

(15  ] 

222122222212222221222 

006 

R 

16 

( 16 

222212222221222222122 

007 

R- 

17 

(17 

222221222222122222212 

008 

R . 

18 

( 18 

222222122222212222221 

009 

R' 

21 

(21 

111111122222222222222 

CIO 

R, 

22 

(22 

322222222222222222222 

01 1 

R / 

2 3 

(23 

232222222222222222222 

012 

R > 

24 

(24 

223222222222222222222 

013 

R / 

25 

(25 

222322222222222222222 

014 

R/ 

26 

(26 

222232222222222222222 

015 

R/ 

27 

(27 

222223222222222222222 

016 

R/ 

28 

(28 

222222322222222222222 

017 

R 

?1 

(31 

222222211111112222222 

018 

R' 

} 2 

(32 

222222232222222222222 

019 

R - 

3 3 

(33 

222222223222222222222 

020 

R ' 

34 

(34 

222222222322222222222 

021 

R- 

3 5 

( 35 

222222222232222222222 

022 

R 

36 

(36 

222222222223222222222 

02  3 

R ’ 

3 7 

' 37 

222222222222322222222 

024 

R ' 

3 P 

(3R 

222222222222232222222 

025 

R / 

41 

(41 

2 22222222222221111111 

026 

R- 

42 

(42 

222222222222223222222 

027 

R 

4 3 

(43 

222222222222222322222 

02  B 

P, 

44 

(44 

222222222222222232222 

029 

R ■ 

45 

(45 

222222222222222223222 

030 

R/ 

46 

(46 

222222222222222222322 

031 

R / 

47 

(47 

222222222222222222232 

03  2 

R. 

48 

(48 

> 222222222222222223223 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  16 

NUMBER  OF  INDEPENDENT  CONTRASTS:  7 


GEND/ 

INTACT : 

1 . 

MALE 

1 . 

NO  MATH 

GEND/ 

INTACT : 

1. 

MALE 

2 . 

8TH  GRD 

GEND/ 

INTACT: 

1. 

MALE 

3 . 

PREALG 

GEND/ 

INTACT: 

1. 

MALE 

4 . 

ALGEBRA 

GEND/ 

INTACT: 

1 . 

MALE 

5. 

INT/SEQ 

GEND/ 

INTACT : 

1 . 

MALE 

6 . 

APPLIED 

GEND/ 

INTACT : 

1 . 

MALE 

7 . 

OTHER 

GEND/ 

INTACT : 

1 . 

MALE 

8. 

MISSING 

GEND/ 

INTACT : 

2 . 

FEMALE 

1 . 

NO  MATH 

GEND/ 

INTACT: 

2. 

FEMALE 

2 . 

RTH  CRD 

GEND/ 

INTACT: 

2. 

FEMALE 

3 . 

PREALG 

GEND/ 

INTACT: 

2. 

FEMALE 

4 . 

ALGEBRA 

GEND/ 

INTACT: 

2 . 

FEMALE 

5. 

INT/SEQ 

GEND/ 

INTACT: 

2. 

FEMALE 

6. 

APPLIED 

GEND/ 

INTACT: 

2. 

FEMALE 

7 . 

OTHER 

GEND/ 

INTACT: 

2 . 

FEMALE 

8 . 

MISSING 

ITY  BY  MATH  COURSES  TAKING  THIS  YEAR 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  32 

NUMBER  OF  INDEPENDENT  CONTRASTS:  21 


RACE/ 

INTACT: 

1 . 

WHI/AI/O 

1 . 

NO  MATH 

RACE/ 

INTACT: 

1 . 

WHI/AI/O 

2. 

8TH  GRD 

RACE/ 

INTACT: 

1. 

WHI/AI/O 

3. 

PREALG 

RACE/ 

INTACT: 

1 . 

WHI/AI/O 

4 . 

ALGEBRA 

RACE/ 

INTACT: 

1 . 

WHI/AI/O 

5. 

INT/SEQ 

RACE/ 

INTACT: 

1 . 

WHI/AI/O 

6. 

APPLIED 

RACE/ 

INTACT: 

1 . 

WHI/AI/O 

7 . 

OTHER 

RACE/ 

INTACT: 

1. 

WHI/AI/C 

8. 

MISSING 

RACE/ 

INTACT: 

2. 

BLACK 

1 . 

NO  MATH 

RACE/ 

INTACT: 

2. 

BLACK 

2 . 

8TH  GRD 

RACE/ 

INTACT: 

2. 

BLACK 

3 . 

PREALG 

RACE/ 

INTACT: 

2 

BLACK 

4 . 

ALGEBRA 

RACE/ 

INTACT: 

2. 

BLACK 

5. 

INT/SEQ 

RACE/ 

INTACT: 

2 . 

BLACK 

6 . 

APPLIED 

RACE/ 

INTACT: 

2. 

BLACK 

7 . 

OTHER 

RACE/ 

INTACT; 

2 . 

BLACK 

8 . 

MISSING 

RACE/ 

INTACT: 

3 . 

HISPANIC 

1 . 

NO  MATH 

RACE/ 

INTACT: 

3. 

HISPANIC 

2 . 

RTH  GRD 

RACE/ 

INTACT: 

3. 

HISPANIC 

3 . 

PREALG 

RACE/ 

INTACT: 

3. 

HISPANIC 

4 . 

ALGEBRA 

RACE' 

INTACT: 

3. 

HISPANIC 

5. 

INT/SEQ 

RACE/ 

INTACT: 

3. 

HISPANIC 

6. 

APPLIED 

RACE/ 

INTACT: 

3 . 

HISPANIC 

7 

OTHER 

RACE/ 

INTACT: 

3. 

HISPANIC 

8 . 

MISSING 

RACE/ 

INTACT: 

4. 

ASIAN 

1 . 

NO  MATH 

RACE/ 

INTACT: 

4 . 

ASIAN 

2 . 

8TH  GRD 

RACE/ 

INTACT: 

4 . 

ASIAN 

3 . 

PREALG 

RACE/ 

INTACT: 

4. 

ASIAN 

4 . 

ALGEBRA 

RACE/ 

INTACT: 

4 . 

ASIAN 

5 . 

INT/SEQ 

PACE/ 

INTACT: 

4. 

ASIAN 

6. 

APPLIED 

RACE' 

INTACT: 

4 . 

ASIAN 

n 

OTHER 

RACE/ 

INTACT: 

4 . 

ASIAN 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES / ASSESSMENTS ■ 
CONDITIONING  VAR  LABEL: 
NAEr  ID: 

TYPE  Or  CONTRAST: 


BACK0062 

INTERACTION:  PARENTS’  EDUCATION  BY  MATH  COURSES  TAKING  THIS  YEAR 

NOR,  SOR 
PARE- 
N.'A 

INTERACTION 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS  - 
NUMBER  OF  INDEPENDENT  CONTRASTS : 


001 

15 

1 1 

( 1 1 

) 33333333333.3  3333333333333333 

PARE/ 

I NTAOT : 

1 . 

« HS 

1 . 

NO  MATH 

no  2 

IV 

12 

(12 

1 122222212222221222222122222 2 

PARE/ 

INTACT; 

1 . 

HS 

2 . 

RTH  GRD 

003 

P/ 

1 3 

(13 

) 2122222212222221222222122222 

PARE/ 

INTACT: 

1 . 

< HS 

3 . 

PREALG 

004 

P 

14 

(14 

1 2212222221222222122222212222 

PARE/ 

INTACT: 

1 . 

HS 

4 . 

ALGEBRA 

005 

P/ 

15 

(15 

) 2221222222122222212222221222 

PARE/ 

I NTACT : 

1 . 

«.  HS 

5. 

INT/SEQ 

006 

r- 

16 

(16 

) 2222122222212222221222222122 

PARE/ 

INTACT: 

1 . 

HS 

6 . 

APPLIED 

007 

p' 

r? 

( 17 

) 2222212222221222222122222212 

PARE/ 

INTACT. 

1 . 

< HS 

7 . 

OTHER 

008 

p ' 

18 

( 18 

1 2222221222222122222212222221 

PARE/ 

INTACT: 

1 . 

HS 

R . 

MISSING 

009 

p. 

21 

(21 

) 1111111222222222222222222222 

PARE/ 

INTACT: 

2. 

HS  GRAD 

1 . 

NO  MATH 

010 

p' 

22 

(22 

) 3222222222222222222222222222 

PARE/ 

INTACT: 

2 . 

HS  GRAD 

2 . 

HTH  GRD 

011 

p 

2 3 

(21 

) 2322222222222222222222222222 

PARE/ 

INTACT: 

2 

HS  GRAD 

3 . 

PREALG 

012 

r 

24 

(24 

) 2232222222222222222222222222 

PARE/ 

INTACT: 

2. 

HS  GRAD 

4 . 

ALGEBRA 

013 

p/ 

, ■ 
j 

(26 

i 2223222222222222222222222222 

PARE/ 

INTACT: 

HS  GPAD 

5. 

INT/SEQ 

014 

p- 

2 6 

(26 

) 2222322222222222222222222222 

PARE/ 

1 NTACT : 

2 . 

HS  GRAD 

6 . 

APPLIED 

015 

r 

7 

t 27 

1 2222232222222222222222222222 

PARE/ 

INTACT: 

2 . 

HS  GRAD 

OTHER 

(i  1 <> 

p 

28 

( 2K 

) 2222223222222222222222222222 

PARE  ' 

INTACT: 

2 . 

HS  GRAD 

H . 

MISSING 

03  7 

p- 

< 1 

( 31 

) 22222221 1 1 11 1 1 22 22 2222222222 

PARE/ 

INTACT: 

3 . 

POST  HS 

1 . 

NO  MATH 

01  H 

r 

(2 

( 

) 2222222322222222222222222222 

PAPE, 

INTACT: 

\ . 

POST  HS 

/>  . 

RTH  GRD 

2W 


337 


019 

P/ 

33 

133 

) 2222222232222222222222222222 

020 

P/ 

34 

(34 

) 2222222223222222222222222222 

021 

P/ 

35 

(35 

) 2222222222322222222222222222 

022 

P/ 

36 

(36 

) 2222222222232222222222222222 

023 

P/ 

37 

(37 

) 2222222222223222222222222222 

024 

P/ 

38 

(38 

) 2222222222222322222222222222 

025 

P/ 

41 

(41 

) 2222222222222211111112222222 

026 

P/ 

42 

(42 

) 2222222222222232222222222222 

027 

P/ 

43 

(43 

} 2222222222222223222222222222 

028 

P.' 

44 

(44 

) 2222222222222222322222222222 

029 

?/ 

45 

(45 

I 2222222222222222232222222222 

030 

P/ 

46 

(46 

) 2222222222222222223222222222 

031 

P / 

47 

(47 

) 2222222222222222222322222222 

032 

P/ 

48 

(48 

} 2222222222222222222232222222 

033 

P/ 

51 

(51 

) 2222222222222222222221111111 

034 

P. 

52 

(52 

) 2222222222222222222223222222 

035 

P/ 

53 

(53 

) 2222222222222222222222322222 

036 

P/ 

54 

(54 

) 2222222222222222222222232222 

037 

P/ 

55 

(55 

) 2222222222222222222222223222 

038 

P.' 

56 

(56 

5 2222222222222222222222222322 

039 

P / 

57 

(57 

) 2222222222222222222222222232 

040 

P/ 

58 

(58 

i 2222222222222222222222222223 

CONDITIONING 

VARIABLE  ID: 

BACK0064 

DESCRIPTION: 

INTERACTION:  TYPE  CF  LOCALE 

GRADES /ASSESSMENTS : 

NO 8.  SCS 

CONDITIONING 

VAR  LABEL: 

TCLS/ 

NAEP  ID 

N A 

TYPE  OF 

CCNT 

RAST: 

INTERACTION 

001  T; 

11 

(11 

3333333333333333333333333333 

002  T / 

12 

(12 

t 1222222122222212222221222222 

003  T/ 

13 

(13 

; 2122222212222221222222122222 

004  T ' 

14 

l 14 

) 2212222221222222122222212222 

005  T" 

1 5 

(15 

i 2221222222122222212222221222 

006  T* 

16 

(16 

i 2222122222212222221222222122 

007  T- 

17 

( 17 

i 2222212222221222222122222212 

008  T' 

18 

(18 

) 2222221222222122222212222221 

C09  T- 

21 

(21 

> 1111111222222222222222222222 

010  T • 

22 

j 22 

) 3222222222222222222222222222 

Oil  T 

23 

(23 

^ 2322222222222222222222222222 

012  T. 

24 

<24 

) 2232222222222222222222222222 

013  T. 

25 

(25 

» 2223222222222222222222222222 

014  T •' 

26 

(26 

i 2222322222222222222222222222 

015  T- 

27 

(2? 

) 2222232222222222222222222222 

016  T 

28 

(28 

l 2222223222222222222222222222 

017  T. 

31 

(31 

i 2222222111111122222222222222 

018  T 

32 

(32 

) 2222222322222222222222222222 

019  T ‘ 

33 

(33 

) 2222222232222222222222222222 

020  T 

34 

<34 

) 2222222223222222222222222222 

021  T 

35 

(35 

) 2222222222322222222222222222 

022  T 

36 

(36 

) 2222222222232222222222222222 

023  T/ 

37 

( 37 

! 2222222222223222222222222222 

024  T 

38 

(36 

> 2222222222222322222222222222 

025  T. 

41 

(41 

\ 2222222222222211111112222222 

026  T/ 

42 

(42 

) 2222222222222232222222222222 

027  T 

43 

(43 

) 2222222222222223222222222222 

028  T, 

44 

(44 

) 2222222222222222322222222222 

02  9 T / 

45 

(45 

) 2222222222222222232222222222 

030  T/ 

46 

<46 

) 2222222222222222223222222222 

031  T' 

47 

(47 

) 2222222222222222222322222222 

032  T / 

48 

(48 

) 222222222222222222223222222 2 

033  T / 

51 

<51 

) 2222222222222222222221111111 

03  4 T, 

52 

<52 

) 2222222222222222222223222222 

035  T/ 

53 

(53 

) 2222222222222222222222322222 

036  T ' 

54 

(54 

1 2222222222222222222222232222 

037  T/ 

55 

(65 

) 2222222222222222222222223222 

038  T ' 

56 

(56 

) 2222222222222222222222222322 

039  T 

57 

( 5"1 

} 2222222222222222222222222232 

C40  T 

5B 

(58 

i 2222222222222222222222222223 

PARE/ 

INTACT : 

3. 

POST  HS 

3. 

PREALG 

PARE/ 

INTACT : 

3 . 

POST  HS 

4 . 

ALGEBRA 

PARE/ 

INTACT : 

3. 

POST  HS 

5. 

INT/SEQ 

PARE/ 

INTACT : 

3. 

POST  HS 

6 . 

APPLIED 

PARE/ 

INTACT: 

3. 

POST  HS 

7 . 

OTHER 

PARE/ 

INTACT : 

3 . 

POST  HS 

0. 

MISSING 

PARE/ 

INTACT: 

4. 

COL  GRAD 

1 . 

NO  MATH 

PARE/ 

INTACT: 

4 . 

COL  GRAD 

2 . 

8TH  GRD 

PARE/ 

INTACT : 

4. 

COL  GRAD 

3. 

PREALG 

PARE/ 

INTACT : 

4. 

COL  GRAD 

4. 

ALGEBRA 

PARE/ 

INTACT : 

4. 

COL  GRAD 

5. 

INT/SEQ 

PARE/ 

INTACT : 

4. 

COL  GRAD 

6. 

APPLIED 

PARE/ 

INTACT : 

4. 

COL  GRAD 

7 . 

OTHER 

PARE/ 

INTACT: 

4 . 

COL  GRAD 

8. 

MISSING 

PARE/ 

INTACT : 

5. 

PARED-? 

1 . 

NO  MATH 

PARE/ 

INTACT : 

5. 

PARED-? 

2. 

8TH  GRD 

PARE/ 

INTACT: 

5. 

PARED-? 

3. 

PREALG 

PARE/ 

INTACT : 

5. 

PARED-? 

4 . 

ALGEBRA 

PARE/ 

INTACT : 

5. 

PARED-? 

5. 

INT/SEQ 

PARE/ 

INTACT: 

5. 

PARED-? 

6. 

APPLIED 

PARE; 

INTACT: 

5. 

PARED-? 

7 . 

OTHER 

PARE,- 

INTACT: 

5. 

PARED-? 

8. 

MISSING 

5 CATEGORIES)  BY  MATH  COURSES  TAKING  THIS  YEAR 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 


NUMBER 

OF  INDEPENDENT  CONTRASTS: 

TOL5  / 

INTACT: 

1. 

BIG 

CTY5 

1 . 

HO  MATH 

TOL5/ 

INTACT: 

1 . 

BIG 

CTY5 

2 . 

8TH  GRD 

TOLS  / 

INTACT: 

1. 

BIG 

CTYS 

3. 

PREALG 

TOL5  / 

INTACT: 

1 . 

BIG 

CTY5 

4 . 

ALGEBRA 

TOLS  / 

INTACT : 

1. 

BIG 

CTY5 

5. 

INT/SEQ 

TOL5  / 

INTACT: 

1 . 

BIG 

CTY5 

6. 

APPLIED 

TOL5  / 

INTACT: 

1. 

BIG 

CTYS 

7 . 

OTHER 

TOL5/ 

INTACT: 

1 . 

BIG 

CTY5 

8. 

MISSING 

TOL5  ' 

INTACT: 

2 . 

MID 

CTY5 

1 . 

NO  MATH 

TOLS . 

INTACT: 

2 . 

MID 

CTY5 

2. 

8TH  GRD 

TOLS; 

INTACT: 

2 

MID 

CTYS 

3 . 

PREALG 

TOLS/ 

INTACT: 

2 „ 

MID 

CTY5 

4 . 

ALGEBRA 

TCL5  - 

INTACT- 

2 . 

MID 

CTY5 

5. 

INT/SEQ 

TOLS 

INTACT: 

2 . 

MID 

CTY5 

6. 

APPLIED 

TOLS/ 

INTACT: 

2. 

MID 

CTY5 

7 . 

OTHER 

TOLS  ' 

INTACT. 

2. 

MID 

CTYS 

8. 

MISSING 

TCL5 

INTACT : 

3 . 

FR.'  BTWN5 

1. 

NO  MATH 

TOLS. 

INTACT: 

3 . 

FR/BTWN5 

2 . 

8TH  GRD 

TOL5 ' 

INTACT: 

3. 

FR/BTWN5 

3. 

PREALG 

TOLS, 

INTACT; 

3 . 

FR/BTWN5 

4 . 

ALGEBRA 

TOLS. 
TOL5 
TOL5 
TCLS  / 
TOL5 , 
TOLS, 
TOLS/ 
TOL5 1 
TOLS '' 
TOLS / 
TOLS  • 
TCL5 
TOLS  / 
TCLS  , 
TOLS 
TOLS 
TOLS 
TOLS 
TCLS 
TOLS, 


INTACT: 
INTACT : 
INTACT: 
INTACT: 
INTACT: 
INTACT : 
INTACT: 
INTACT : 
INTACT : 
INTACT : 
INTACT : 
INTACT: 
INTACT: 
INTACT: 
INTACT : 
INTACT : 
INTACT: 
INTACT : 
INTACT : 
INTACT: 


FR/BTWN5  5 
FR/BTWN5  6 
FR/BTWN5  7 
FR/BTWN5  0 
SML  TWN5  1 
SML  TWN5  2 
SML  TWN5 
SML  TWN5 
SML  TWN5 
SML  TWNS 
SML  TWNS 
SML  TV/NS 
RURAL 5 
RURALS 
RURAL5 
RURALS 
RURALS 
RURALS 


5 . RURAL5 


INT/SEQ 

APPLIED 

OTHER 

MISSING 

NO  MATH 

8TH  GRD 

P REA LG 

ALGEBRA 

INT/SEQ 

APPLIED 

OTHER 

MISSING 

NO  MATH 

8TH  GRD 

PREALG 

ALGEBRA 

INT'SEQ 

APPLIED 

OTHER 

MISS  TNG 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK006  6 

INTERACTION : SCHOOL  TYPE  3Y  MATH  COURSES  TAKING  THIS  YEAR 

NO ft,  SO 8 
SCHT ' 

N A TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

INTERACTION  NUMBER  OF  INDEPENDENT  CONTRASTS: 


24 

14 


001 

S/ 

11 

(11 

) 010101010 101 01 010 10101C 10101 

SCHT' 

INTACT 

1 . 

PUBLIC 

1 . 

NO  MATH 

002 

s 

12 

i 12 

) -1000000000000-1000000000000 

SCHT  t 

INTACT 

1 . 

PUBLIC 

2 . 

8TH  GRD 

003 

s. 

1 3 

{ 1 * 

) 0n~ 1 000900000000-10000000000 

SCHT/ 

INTACT 

1. 

PUBLIC 

3 . 

PREALG 

004 

14 

< 14 

> 0000-1000000000000  - 1 oooncooo 

SCHT  ' 

INTACT 

1 . 

PUBLIC 

4 . 

ALGEBRA 

005 

s* 

IS 

<15 

) r 00000- 1 000000000000-1000000 

SCHT 

INTACT 

1 . 

PUBLIC 

5. 

I NT/ 5EQ 

006 

S/ 

16 

(16 

) 00000000-1000000000000- 10000 

SCHT/ 

INTACT 

1 . 

PUBLIC- 

6 . 

APPLIED 

007 

s. 

17 

( 17 

) 0000000000' 1000000000000-100 

SCHT/ 

INTACT 

1 . 

PUBLIC 

7 . 

OTHER 

3(H) 


008 

S/ 

18 

(18 

009 

S/ 

21 

(21 

010 

S/ 

22 

(22 

Oil 

S/ 

23 

(23 

012 

S/ 

24 

(24 

013 

S/ 

25 

(25 

014 

S/ 

26 

(26 

015 

S/ 

27 

(27 

016 

S/ 

28 

(28 

017 

Si 

31 

(31 

018 

S/ 

32 

(32 

019 

S/ 

33 

(33 

020 

S/ 

34 

(34 

021 

S/ 

35 

(35 

022 

S/ 

36 

(36 

023 

S/ 

37 

(37 

024 

S/ 

38 

(38 

) 000000000000-1000000000000-1 
) -1-1-1-1-1-1-100000000000000 
) 0100000000000000000000000000 
) 0001000000000000000000000000 
) 0000010000000000000000000000 
) 0000000100000000000000000000 
) 0000000001000000000000000000 
) 0000000000010000000000000000 
) 0000000000000100000000000000 
) 00000000000000-1-1-1-1-1-1-1 
) 0000000000000001000000000000 
) 0000000000000000010000000000 
) 000000000000000'-  QOIOGOOOOOO 
) 000000000000000c  . '0001000000 
} 0000000000000000000000010000 
) 0000000000000000000000000100 
) 0000000000000000000000000001 


SCHT/ 

INTACT: 

1 . 

PUBLIC 

8. 

MISSING 

SCHT/ 

INTACT : 

2. 

PRIVATE 

1. 

NO  MATH 

SCHT/ 

INTACT: 

2. 

PRIVATE 

2. 

8TH  GRD 

SCHT/ 

INTACT: 

2 . 

PRIVATE 

3 . 

PREALG 

SCKT/ 

INTACT: 

2. 

PRIVATE 

4. 

ALGEBRA 

SCHT/ 

INTACT: 

2. 

PRIVATE 

5. 

INT/SEQ 

SCHT/ 

INTACT : 

2 . 

PRIVATE 

6. 

APPLIED 

SCHT/ 

INTACT : 

2. 

PRIVATE 

7 . 

OTHER 

SCHT/ 

INTACT : 

2. 

PRIVATE 

B. 

MISSING 

SCHT/ 

INTACT: 

3 . 

CATHOLIC 

1 . 

NO  MATH 

SCHT/ 

INTACT: 

3. 

CATHOLIC 

2. 

8TH  GRD 

SCHT/ 

INTACT: 

3. 

CATHOLIC 

3. 

PREALG 

SCHT/ 

INTACT: 

3 . 

CATHOLIC 

4 . 

ALGEBRA 

SCHT/ 

INTACT: 

3 . 

CATHOLIC 

5. 

INT/SEQ 

SCHT/ 

INTACT : 

3 . 

CATHOLIC 

6. 

APPLIED 

SCHT/ 

INTACT: 

3 . 

CATHOLIC 

7 . 

OTHER 

SCHT/ 

INTACT: 

3 . 

CAHOLIC 

8. 

MISSING 

CONDITIONING  VARIABLE  ID:  BACK0068 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


MSA/ NON -MSA 
S04 , SOB 
MSATSA 
MA93FLG 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  MSA  {0 

002  NON  MSA  (1 

003  MSA-MISS  (M 


MSA 

NON -MSA 
MSA  MISS TNG 


CONDITIONING  VARIABLE  ID:  BACK0069 

DESCRIPTION:  STATE  AD 

GRADES/ASSESSMENTS:  S04  , SOB 

CONDITIONING  VAR  LABEL:  MONITOR 

NAEP  ID:  MONSTUD 

TYPE  OF  CONTRAST:  CLASS 


STATE  ADMINISTRATION  MONITORED /UNMONITORED  SESSION 

S04 , SOB 

MONITOR 

MONSTUD  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS. 


001  UNMONIT  (0 

002  MONITOR  (1 


UNMONITORED  SESSION 
MONITORED  SESSION 


CONDITIONING  VARIABLE  ID:  BACK0070 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


INTERACTION : 
S04 , S08 
SCHT/MON 
N/A 

INTERACTION 


SCHOOL  TYPE  BY  MONITORED/UNMONITORED  SESSION 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


S/M 

11 

(11 

) 0101 

SCHT/MON 

INTACT 

1 . 

PUBLIC 

1 . 

UNMONIT 

S/M 

12 

(12 

) - 1-1 

SCHT/MON 

INTACT 

1 . 

PUBLIC 

2 

MONITOR 

S/M 

21 

(21 

) -100 

SCHT/MON 

INTACT 

2 . 

PRIVATE 

1 . 

UNMON I T 

S/M 

22 

(22 

; oioo 

SCHT/MON 

INTACT 

2 . 

PRIVATE 

2 . 

MONITOR 

S/M 

31 

(31 

) 00-1 

SCHT/MON 

INTACT 

3 . 

CATHOLIC 

1 . 

UNMONIT 

S/M 

32 

(32 

) 0001 

SCHT ' MON 

INTACT 

3 . 

CATHOLIC 

2 . 

MONITOR 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0071 

SAMPLE  TYPE 

N04 , S04 , NOB.  SOB.  N1 

SUBSAMP 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  SAMP  SI  (01 

002  SAMP  S2  (02 

003  SAMP  S3  (03 


SAMPLE  SI 
SAMPLE  S2 
SAMPLE  S3 


CONDITIONING  VARIABLE  ID:  BACK0072 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 


INTERACTION:  SAMPLE  BY  RACE/ ETHNICITY 

N04 , S04 , NOB,  S08,  N12 


CONDITIONING 

VAR  LABEL: 

/RAC 

NAEP 

ID: 

N/A 

TYPE 

OF 

CONTRAST : 

INTERACTION 

001 

/R 

11 

(11 

) 010101010101 

002 

/R 

12 

(12 

) -10000-10000 

003 

/R 

13 

(13 

) 00-10000-100 

004 

/R 

14 

(14 

) 0000-10000-1 

005 

/R 

21 

(21 

) -1-1-1000000 

006 

/R 

22 

(22 

) 010000000000 

007 

/R 

23 

(23 

) 000100000000 

00R 

/R 

24 

(24 

> 000001000000 

009 

/K 

n 

( 11 

) 000000-1-1-1 

010 

/R 

12 

( 12 

) 000000010000 

Oil 

/R 

33 

(33 

) 000000000100 

TOTAL  NUMBER  OF  SPECIFIED  CONTRA..  S 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


/RAC 

INTACT 

1 . 

SAMP 

SI 

1 . 

WHI/AI '0 

/RAC 

INTACT 

1 . 

SAMP 

SI 

2 . 

BLACK 

/RAC 

INTACT 

1 . 

SAMP 

SI 

3 . 

HISPANIC 

/RAC 

INTACT 

1 . 

SAMP 

SI 

4 . 

ASIAN 

/RAC 

INTACT 

2. 

SAMP 

G2 

1 . 

WHI /AI 10 

'RAC 

INTACT 

O 

c.  * 

SAMP 

S2 

2. 

BLACK 

/RAC 

INTACT 

2 . 

SAMP 

S2 

3 . 

HISPANIC 

/RAC 

INTACT 

2 . 

SAMP 

S2 

4 . 

ASIAN 

/FAC 

I NTACT 

3 . 

SAMP 

S3 

1 . 

WHI / Al /U 

/RAC 

INTACT 

3 . 

SAMP 

S3 

BLACK 

/RAC 

INTACT 

3 . 

SAMP 

S3 

3 . 

HISPANIC 

339  BEST  COPY  AVAILABLE 


012  /R  34  (34  ) 

000000000001 

/RAC  INTACT:  3.  SAMP  S3  4.  ASIAN 

CONDITIONING  VARIABLE  ID: 

BACK007  3 

DESCRIPTION: 

REPORTING  SAMPLE 

GRADES /ASSESSMENTS : 

N04,  S04,  NOR,  SOB,  N12 

CONDITIONING  VAR  LABEL: 
NAEP  ID. 

RPTSAMP 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  RPTSAMP  (01 

> o 

YES 

002  RPT  NO  (02 

) 1 

NO 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

BACK0074 
HOW  OFTEN 

USE  A HOME  COMPUTER 

FOR  SCHOOLWORK? 

GRADES /ASSESSMENTS : 

N04 , S04 , 

NOR,  S08 , N 12 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

B00930I 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  B00930IA  (01 

00000 

ALMOST  EVERY  DAY 

002  B009301R  (02 

10000 

ONCE  OR  TWICE  A WEEK 

001  BO09301C  (03 

01000 

ONCE  OR  TWICE  A MTH 

004  B009301D  (04 

00100 

NEVER  OR  HARDLY  EVER 

OC'  B009301E  (OS 

00010 

NO  COMPUTER  AT  HOME 

0 06  P.009  301  M (M 

00001 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION. 

GRADES 'ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0075 

HOW  SAFE  DO  YOU  FEEL  AT  SCHOOL"5 
N04 , S04,  NOB,  S08 , N12 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


B009401 

CLASS 


00 J B00940JA  (01  ) 0000 

002  BO 09  40  IB  (02  ) 1000 

no  3 B000401C  (03  ) 0)00 

004  BO 0940 1 D (04  1 0010 

•■OS  P, 00940 1M  (M  ) 0001 


VERY  SAFE 
SOMEWHAT  SAFE 
SOMEWHAT  UNSAFE 
VERY  UNSAFERDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

IV PE  OF  CONTRAST : 


BACK0076 

DOES  YOUR  STEP /'MOTHER  WORK  AT  A JOB  FOR  PAY"5 
N04 , SO 4 

B009S0 1 TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


■ ni  P.009  SO  1 A (01  ) 000 

00..  BO 09 SO  1 R (02  ) 100 

Om  H009S01C  (03  ) 010 

004  PU09S01M  (M  ) 001 


YES 

NO 

DON'T  LIVE  W/  MOTHER 
MISSING 


fnNDl TIONING  VARIABLE  ID: 
DESCRIPTION; 

< GRADES < ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID- 

TYPE  OF  CONTRAST: 


BACK0077 

DOES  YOUR  STEP/ 1 VTHER  WORK  AT  A JOB  FOR  TAY? 

N04.  S04 

B009S02  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CIiASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


i wi|  BOO 9 B 02 A (01  1 000 

on;1  P009S02B  (02  ) 100 

001  ROO’C.  VC  l .'l  ) 010 

(iici  (m  > 001 


YES 

NO 

DON'T  LIVE  W / FATHER 
MISSING 


conditioning  variable  id: 
description. 

ORADPA'L  ASSE.SSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID. 

TYPE  <>K  CONTRAST: 


BACK007H 

DESCRIBE  YOUR  OVERALL  GRADES  SINCE  6TH  GRADE 
NOR,  SOB 

B00970 1 TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


on  i 

BU097  0 1 A 

(01 

000000 

002 

BOO 9 70 1 B 

( 2 

100000 

on  * 

boo  >->nic 

((<  ’ 

010000 

no  4 

BOO 9 / Oil) 

(04 

001000 

IhV, 

BOOM  Hi  1 F 

((>'• 

000100 

Ih-t. 

MOO’l  /OIK 

( (Hi 

000010 

{ <4  . 1 

JOB;'*  7 0 1 M 

(M 

000001 

■ ■ >NI 

» m oh  i 14' ; 

VARIABLE  ID: 

BACKOOV9 

l iF.:‘ 

'KIPTION. 

HOW  FAR  IN 

MOSTLY  A 1 S 
MOSTLY  B'S 
MOSTLY  C'S 
MOSTLY  D'S 
MOSTLY  BELOW  D'S 
CLASSES  NOT  GRADED 
MISSING 


SCHOOL  DO  YOU  THINK  YOU  WILL  GO*' 


302 


• -T 

O *i  ! 


BEST  COPY  AVAILABLE 


GRADES /ASSES  SMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


B009801 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  NO  HS  (01 

002  GRAD  HS  (02 

003  ED  > HS  (03 

004  GRAD  CLG  (04 

005  GRAD  SCH  (05 

006  IDK  (06 

007  MISSING  (M 


) 000000 
) 100000 
) 010000 
) 001000 
) 000100 
) 000010 
) 000001 


NOT  FINISH  HS 
GRADUATE  HS 
SOME  ED  PAST  HS 
GRADUATE  COLLEGE 
GO  GRADUATE  SCHOOL 
I DON'T  KNOWGRADED 
MISSING 


CONDITIONING  VARIABLE  ID:  BACK0080 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


DOES  YOUR  STEP/MOTHER  WORK  AT  A JOB  FOR  PAY? 
N08 , S08 , N12 


B009601 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  B009601A  (01 

002  B009601B  (02 

003  B009601C  (03 

004  B009601D  (04 

005  B009601M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


YES,  FULL-TIME 
YES,  PART-TIME 
NO 

DON'T  LIVE  W/  MOTHER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST : 


BACK0081 

DOES  YOUR  STEP/ FATHER  WORK  AT  A JOB  FOR  PAY? 
N08 , S08 , N12 


B009602 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  B009601A  (01 

002  B009601B  (02 

003  B009601C  (03 

004  B009601D  (04 

005  E009601M  (M 


YES,  FULL-TIME 
YES,  PART-TIME 
NO 

DON'T  LIVE  W/  FATHER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BACK0090 

FIPS  CODE  FOR  THE  STATE  DATA 
S04 , S08 

FIPS 

LINEAR 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


CONDITIONING  VARIABLE  ID:  SUBJ0001 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


WHAT  KIND  OF  MATH  CLASS  ARE  YOU  TAKING  THIS  YEAR? 
N08 , S08 


MATTAKE 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


NO  MATH 

8TH  GRD 

PREALG 

ALGEBRA 

INT/SEQ 

APPLIED 

OTHER 

MISSING 


) 0000000 
) 1000000 
) 0100000 
) 0010000 
) 0001000 
) 0000100 
) 0000010 
) 0000001 


NOT  TAKING  THIS  YEAR 
EIGHTH  GRADE  MATH 
PREALGEBRA 
ALGEBRA 

INTEGRATED /SEQUENTAL 
APPLIED  MATH 
OTHER  MATH  CLASSUR 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0002 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  DO  YOU  DO  MATH  PROBLEMS  FROM  TEXTBOOK? 
N04,  SO  4 , N08,  S08,  N12 


M812701 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812701A  (01 

002  M812701B  (02 

003  M812701C  (03 

004  M812701D  (04 

005  M812701M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 


SUB J 0003 

HOW  OFTEN  DO  YOU  DO  MATH  PROBLEMS  ON  WORK  SHEETS  ? 
N04 , S04 , N08,  S08,  N12 


7 

6 


5 

4 


5 

4 


1 

1 


8 

7 


5 

4 


BEST  COPY  AVAILABLE 


303 


NAEP  ID: 

M812702 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  M8I2702A 

(01  ) 

0000 

ALMOST  EVERY  DAY 

002  M812702B 

(02  ) 

1000 

ONCE  OR  TWICE  A WEEK 

003  M812702C 

(03  ) 

0100 

ONCE  OR  TWICE  MONTH 

004  M812701D 

(04  ) 

0010 

NEVER  OR  HARDLY  EVER 

005  M812701M 

<M  ) 

0001 

MISSING 

CONDITIONING 

VARIABLE  ID: 

SUBJ0004 

DESCRIPTION: 

HOW  OFTEN 

SOLVE  MATH  PROBLEMS 

WITH  PARTNER /GROUP? 

GRADES /ASSESSMENTS : 

N04 , S04 , 

N08,  SO  8 , N12 

CONDITIONING 

VAR  LABEL: 

NAEP  ID: 

M812703 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  M812703A 

(01 

) 0000 

ALMOST  EVERY  DAY 

002  M812703B 

(02 

i 1000 

ONCE  OR  TWICE  A WEEK 

003  M812703C 

(03 

) 0100 

ONCE  OR  TWICE  MONTH 

004  M812703D 

(04 

) 0010 

NEVER  OR  HARDLY  EVER 

005  M812703M 

(M 

) 0001 

MISSING 

CONDITIONING 

VARIABLE  ID: 

SUBJ0005 

DESCRIPTION: 

HOW  OFTEN 

IN  MATH  WORK  W/  RULERS,  GEOM  SHAPES? 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


N04,  S04 


M812704 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

M812704A 

(01 

) 0000 

002 

M812704B 

(02 

) 1000 

003 

M812704C 

(03 

) 0100 

004 

M812704D 

(04 

) 0010 

005 

M812704M 

(M 

) 0001 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0006 

HOW  OFTEN  WRITE  ABOUT  SOLVING  A MATH  PROBLEM? 

N04 , S04 , N08 , SOB.  N12 

M8127Q5  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

M812705A 

(01 

) 0000 

002 

M812705B 

(02 

) 1000 

003 

M812705C 

(03 

} 0100 

0C4 

M812705D 

(04 

) 0010 

005 

M812705M 

(M 

) 0001 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING  RDLY  EVER 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0007 

HOW  OFTEN  DO  YOU  TAKE  MATHEMATICS  TESTS? 

N04 . S04 , NOB,  S08,  N12 

M812706  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812706A  (01  ) 0000 

002  M812706B  (02  ) 1000 

003  M812706C  (03  ) 0100 

004  M812706D  (04  ) 0010 

C05  MR12706M  (M  ) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


5 

4 


5 

4 


5 

4 


5 

4 


5 

4 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0008 

HOW  OFTEN  DO  YOU  TALK  TO  THE  CLASS  RE  MATH  WORK? 

N04 , S04 , N08 , S08,  N12 

M812707  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812707A  (01  ) 0000 

002  M812707B  (02  ) 1000 

003  M812707C  (03  ) 0100 

004  MR12707D  (04  ) 0010 

005  MR12707M  (M  ) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


5 

4 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0009 

HOW  OFTEN  DO  10  OR  MORE  MATH  PROBLEMS  BY  YOURSELF? 

N04.  S04 , NOB.  S08,  N12 

M8 12708  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS- 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


304 


001  M8 12708A  (01 

002  M812708B  (02 

003  M8127G8C  (03 

004  M812708D  (04 

005  M812708M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0010 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  M812709A  (01 

002  M812709B  (02 

003  M812709C  (03 

004  M812709D  (04 

005  M812709M  (M 


HOW  OFTEN  DISCUSS  MATH  PROBLEMS  W/  OTHER  STUDENTS? 
N04,  S04 , NOS , S08 , N12 


M812709 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0011 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  M812710A  (01 

002  M81271QB  (02 

003  M812710C  (03 

004  M812710D  (04 

005  M812710M  (M 


HOW  OFTEN  DO  YOU  USE  A COMPUTER  FOR  MATH  ? 
N04 , S04 , NOS , S08 , N12 


M812710 

CLASS 

) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0012 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  DO  YOU  USE  A CALCULATOR  FOR  MATH? 
NO 4 , S04 , N08 , S08,  N12 


M812711 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812711A  (01 

002  M812711B  (02 

003  M812711C  (03 

004  M8 1 27 1 ID  (04 

005  M812711M  (M 


} 0000 
) 1000 
) 0100 
) 0010 
) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0013 


DESCRIPTION: 

GRADES  ./ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


DO  YOU  HAVE  CALCULATOR  TO  USE  WITH  MATH  HOMEWORK? 
N04 , S04 , NOB,  S08,  N12 


Mfil 1201 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M811201Y  (01 

002  M8U201N  (02 

003  MR11201M  (M 


YES 

NODECIDED 

MISSING 


CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES/ ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  USE  A CALCULATOR  FOR  MATH  CLASSWORK? 
N04 , S04 , NOR,  S08,  N12 


M812001 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812001A  (01 

002  M8120C1B  (02 

003  M812001C  (03 

004  M812001D  (04 

005  MR12001M  (M 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0015 


DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  USE  A CALCULATOR  FOR  MATH  HOMEWORK? 
N04,  S04 , NOB,  S08,  N12 


M812002 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812002A  (01 

002  MB  1 2002 B (02 

003  MB  12 00 1C  (03 

004  M812001D  (04 

005  MR12001M  (M 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 
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BEST  COPY  AVAILABLE 


CONDITIONING  VARIABLE  ID:  SUBJ0016 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  USE  A CALCULATOR  FOR  MATH  TESTS/QUIZZES? 
N04,  S04,  N08,  S08,  N12 


M812003 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812003A  {01 

002  M812003B  (02 

003  M812003C  (03 

004  M812003D  (04 

005  M812003M  (M 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0017 


DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  MUCH  TIME  SPEND  DAILY  ON  MATH  HOMEWORK? 
N04 , S04 , N08 , S08,  N12 


M812101 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


M812101A  (01 
M812101B  (02 
M0121O1C  (03 
M812101D  (04 
M812101E  (05 
M812101F  (06 
M812101G  (07 
M812101M  (M 


) 0000000 
) 1000000 
) 0100000 
) 0010000 
) 0001000 
) 0000100 
) 0000010 
) 0000001 


NOT  TAKING  THIS  YEAR 
NONE 

15  MINUTES 
30  MINUTES 
45  MINUTES 
ONE  HOUR 

MORE  THAN  ONE  HOUR 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0018 

GET  MATH  HELP  FROM  SPECIAL  TEACHER/ AIDE/TUTOR? 
N04 , S04 


M811401 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M811401Y  (01 

002  M811401N  (02 

003  M811401M  (M 


YES 

NODECIDED 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0019 

AGREE/DISAGREE:  I LIKE  MATH 
N04 , S04 

M811101 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M811101A  (01 

002  M811101B  (02 

003  M811101C  (03 

004  MRU101M  (M 


AGREE 
UNDECIDED 
DISAGREE  OR  LESS 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0020 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


AGREE/ DISAGREE:  I AM  GOOD  AT  MATH 

N04,  S04 


M811103 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M8U103A  (01 

002  M811103B  (02 

003  M811103C  (03 

004  M811103M  (M 


AGREE 
UNDECIDED 
DISAGREE  OR  LESI 
'1ISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0021 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  MUCH  AGREE- UNDERSTAND  MOST  OF  MATH  CLASS? 
NO 4 , 304 


MB11106 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M8111C6A  (01 

002  M811106B  (02 

003  M8U106C  (03 

004  M811106M  (M 


AGREE 
UNDECIDED 
DISAGREE  OR  LESS 
MISSING 


CONDITIONING  VARIABLE  ID:  3UBD0022 

DESCRIPTION:  HOW  MUCH  AGREE -ONLY  1 CORRECT  WAY 

GRADES/ASSESSMENTS:  N04,  S04 
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/*  /< 

U*?  J 


CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST; 


M811109 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


4 

3 


001  M811109A  {01 

002  Meill09B  (02 

003  M311109C  (03 

004  M811109M  (M 


AGREE 
UNDECIDED 
DISAGREE  OR  LESS 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0023 


DESCRIPTION: 

GRADES, 'ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  MUCH  AGREE- LEARNING  MATH  IS  MEMORIZING  FACTS? 
N04,  S04 


M811107 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M811107A  (01 
0C2  M81UG7B  (02 
003  M811107C  (03 
0C4  M811107M  (M 


AGREE 
UNDECIDED 
DISAGREE  OR  LESS 
MISSING 


CCNDITIONING  VARIABLE  ID:  SUBJ0024 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


AGREE/ DISAGREE:  MATH  USED  FOR  SOLVING  PROBLEMS 
N04,  SC4 


M8I1105 

CLASS 


TOTAL  NUM3ER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  K811105A  (01 

002  M.811105B  (02 

003  M811305C  (03 

004  K81I105M  (M 


AGREE 
UNDECIDED 
DISAGREE  OR  LESS 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0025 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  CF  CONTRAST: 


HOW  MUCH  AGREE- IF  CHOICE  I WOULDN'T  STUDY  MORE7 
N04 , S04 


M81110R 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS. 


0C1  M811108A  (01 
002  K811108B  (02 
C03  M811106C  (03 
004  M811108M  (M 


AGREE 
UNDECIDED 
DISAGREE  CR  LESS 
MISSING 


CCNDITIONING  VARIABLE  ID:  SUBJC02 6 


DESCRIPTION: 

GRADES ; ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID. 

TYPE  OF  CONTRAST- 


HOW  MUCH  AGREE- EVERYONE  CAN  DO  WELL  IF  TRY7 
NO 4 , S04 


M811110 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


201  M8U110A  f 01 
002  M811110B  (02 
0C3  MR  11 1 20C  (03 
004  M81111CM  (M 


AGREE 

undecided 

DISAGREE  OR  LESS 
MISSING 


CONDITIONING  VARIABLE  ID-  SUBJ002” 


DESCRIPTION: 

GRADES . ASSESSMENTS . 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CCNTRAS: 


ABOUT  HOW  MANY  QUESTIONS  RIGHT  ON  TEST7 
N04,  S04 , NOB.  S08 . NI2 


MM00101 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  CF  INDEPENDENT  CONTRASTS: 


001  MKD0101A  (01 

002  KM0C-101B  (02 
CO 3 MM201C1C  (03 
0C4  MMC0101D  (04 
005  KM001CIM  (M 


ALMOST  ALL 
MORE  THAN  HALF 
ABOUT  HALF 
LESS  THAN  HALF 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJOC'28 


DESCRIPTION: 

GRADES . ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEF  ID: 

TYPE  OF  CONTRAST: 


HOW  HARD  TEST  COMPARED  TO  TH"‘SE  IN  SCHOOL' 
N04 , S04 , NOB,  S08.  N12 


MM00201 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRAST: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


r'l  MMOO? 0 1 A 

««1 

1 0000 

MUCH  HAPDEP 

•-2  MM  OOP  0 IB 

i ijt 

) 10C0 

HARDER  THAI  OTHER 

07  MM" 020 1C 

( 0 l 

) Ox  CO 

ABOUT  THE  SAME 

?4  MM00201D 

(04 

) 0010 

EASIEP  THEN  DTHES 

nCDT 


rnDV  AVAILABLE 


o a rr 
u ‘ J J 


005  MM00201M  (M 


) 0001 


MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0029 

HOW  HARD  DID  YOU  TRY  ON  TEST  COMPARED  TO  OTHERS? 

N04,  S04 , N08 , S08,  N12 

MM0030 1 TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001 

MM00301A 

(01 

) 0000 

MUCH  HARDER 

002 

MM00301B 

(02 

) 1000 

HARDER  THAN  OTHERS 

003 

MM00301C 

(03 

) 0100 

ABOUT  AS  HARD 

004 

MM00301D 

(04 

) 0010 

NOT  AS  HARD  AS  OTHER 

005 

MM00301M 

(M 

) 0001 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ003  0 

HOW  IMPORTANT  WAS  IT  YOU  DO  WELL  ON  THIS  TEST? 

N04 , S04 , NOS,  S08 , N12 

MM00401  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001  MM00401A  (01  ) 0000 

002  MM00401B  (02  ) 1000 

003  MM00401C  (03  ) 0100 

004  MM00401D  (04  ) 0010 

005  MM00401M  (M  ) 0001 


VERY  IMPORTANT 
IMPORTANT 

SOMEWHAT  IMPORTANT 
NOT  VERY  IMPORTANTER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ003 1 

HOW  OFTEN  HAD  TO  WRITE  LONG  ANSWERS  TO  QSTS? 

NC4 , S04 , N08 , S08,  N12 

MM00501  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  MM00501A  (01  ) 0000 

002  MM00501B  (02  ) 1000 

003  MM00501C  (03  ) 0100 

004  MM00501D  (04  ) 0010 

005  MM00501M  (M  ) 0001 


AT  LEAST  ONCE  A WEEK 
ONCE  OR  TWICE  A MNTH 
ONCE  OR  TWICE  A YEAR 
NEVERERY  IMPORTANTER 
MISSING 


5 

4 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0032 

WORK  W/  MEAS.  INSTRUMENTS /GEOM  SOLIDS  FOR  MATH? 

N08,  SOS,  N12 

M812712  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

M812712A 

(01 

) 0000 

002 

M812712B 

(02 

) 1000 

003 

M812712C 

(03 

) 0100 

004 

M812712D 

(04 

) 0010 

005 

M812712M 

(M 

) 0001 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


5 

4 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0033 

HOW  OFTEN  WRITE  REPORTS  OR  DO  MATH  PROJECTS? 

NO 8,  SO 8 , N12 

M812713  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001  M812713A  (01  } 0000 

002  M812713B  (02  ) 1000 

003  M812713C  (03  ) 0100 

004  M812713D  (04  ) 0010 

005  M812713M  (M  ) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING  RDLY  EVER 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0034 

IS  THERE  A PORTFOLIO  W/  YOUR  MATH  WORK  IN  IT? 

N08,  SOR,  N12 

M812201  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

MR12201Y 

(01 

) 00 

YES 

002 

MB12201N 

(02 

) 10 

NONE 

003 

M ft  1 2 2 0 1 M 

(M 

) 01 

MISSING 

3 

2 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 


SUBJ00  35 

DO  YOU  USE  A SCIENTIFIC  CALCULATOR  FOR  MATH  WORK? 


308 


U H \ 


GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


NOR , SOS , N12 


M812301 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812301Y  (01 

002  M812301N  (02 

003  M812301M  (M 


YES 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0036 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


DO  YOU  USE  A GRAPHING  CALCULATOR  FOR  MATH  WORK? 
N08 , S08 , N12 


M812401 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812401Y  (01 

002  M81240 IN  (02 

003  M812401M  (M 


YES 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0037 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


WHAT  KIND  OF  MATH  CLASS  WILL  YOU  TAKE  IN  9TH  GR? 
N08 , S08 


MAT EXP 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


NO  MATH 

BASIC 

APPLIED 

PREALG 

ALG  1 

GEOMETRY 

INT/SEQ 

OTHER 

IDK 

MISSING 


) 000000000 
) 100000000 
) 010000000 
) 001000000 
) 000100000 
) 000010000 
) 000001000 
) 000000100 
> 000000010 
) 000000001 


NOT  EXPECT  TO  TAKE 
BASIC , GEN , BUSINESS 
APPLIED  MATH 
PREALGEBRA 
ALGEBRA  1/ELEM  ALG 
GEOMETRY 

INTERGATED/ SEQUENTAL 
OTHER  MATH  CLASS 
I DON'T  KNOW 
MISSING  W 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SUBJ0038 

DO  YOU  AGREE:  I LIKE  MATH 
N08 , S08 , N12 

M810701 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M812701A  (01 

002  M812701B  (02 
00*  M812701C  (03 

004  M812701D  (04 

005  M812701E  (05 

006  M812701M  (M 


) 00000 
) 10000 
) 01000 
) 00100 
) 00010 
) 00001 


STRONGLY  AGREE 
AGREE 
UNDECIDED 
DISAGREE 

STRONGLY  DISAGREE 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST; 


SUBJ0039 

DO  YOU  AGREE:  I AM  GOOD  IN  MATH 
N08 , SOB,  N12 

M810703 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M810701A  (01 

002  M810701B  (02 

003  M810701C  (03 

004  M8 1 070 ID  (04 

005  M810701E  (05 

006  M810701M  (M 


STRONGLY  AGREE 
AGREE 
UNDECIDED 
DISAGREE 

STRONGLY  DISAGREE 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0040 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


AGREE /DISAGREE:  UNDERSTAND  MOST  OF  MATH  CLASS 
N08 , S08 , N12 


M810707 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  M810707A  (01 

002  M810707B  (02 

003  M810707C  (03 

004  MB10707D  (04 

005  M810707E  (05 

006  MR10707M  (M 


) 00000 
) 10000 
) 01000 
) 00100 
) 000  L0 
) 00001 


STRONGLY  AGREE 
AGREE 
UNDECIDED 
DISAGREE 

STRONGLY  DISAGREE 
MISSING 


CONDITIONING  VARIABLE  ID:  SUBJ0041 


34? 


* m 


m%  . m •«  mi  r* 


DESCRIPTION: 


AGREE/ DISAGREE:  ONLY  ONE  WAY  TO  SOLVE  MATH  PROBLEM 


GRADES /ASSESSMENTS : 

N08 , S08 , N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

M810709 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  M8107U9A  (01  ) 

00000 

STRONGLY  AGREE 

002  M810709B  (02  1 

10000 

AGREE 

003  M810709C  (03  ) 

01000 

UNDECIDED 

004  MB10709D  (04  ) 

00100 

DISAGREE 

005  M810709E  (05  ) 

00010 

STRONGLY  DISAGREE 

006  M810709M  (M  ) 

00001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SUBJ0042 

DESCRIPTION: 

AGREE /DISAGREE:  MATH 

IS  MOSTLY  MEMORIZING  FACTS 

GRADES/ ASSESSMENTS : 

N08,  S08 , N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

M810708 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  M810708A  (01  ) 

00000 

STRONGLY  AGREE 

002  M810708B  (02  ) 

10000 

AGREE 

003  M810708C  (03  ) 

01000 

UNDECIDED 

004  M810708D  (04  ) 

00100 

DISAGREE 

005  M810708E  (05  ) 

00010 

STRONGLY  DISAGREE 

006  M810708M  IM  ) 

00001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SUBJ0043 

DESCRIPTION: 

AGREE /DISAGREE:  CONCEPTS 

ARE  AS  IMPORTANT  AS  OPER 

GRADES/ ASSESSMENTS : 

N08 , S08 , N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

M810710 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  M810710A  (01  ) 

i 00000 

STRONGLY  AGREE 

002  M810710B  (02  ) 

> 10000 

AGREE 

003  M8 107 IOC  (03  1 

1 01000 

UNDECIDED 

004  M810710D  (04  ] 

l 00100 

DISAGREE 

005  M810710E  (05  1 

> 00010 

STRONGLY  DISAGREE 

006  M610710M  <M 

1 00001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SUBJ0044 

DESCRIPTION: 

DO  YOU  AGREE:  MATH  USEFU 

L/ SOLVING  EVERYDAY  PROBLEM 

GRADES /ASSESSMENTS : 

NOB,  SOR,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

M810705 

TOT/XL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST : 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  M810705A  (01 

) 00000 

STRONGLY  AGREE 

002  MR 10705B  (02 

) 10000 

AGREE 

003  M8107C5C  (03 

} 01000 

UNDECIDED 

004  M810705D  (04 

) 00100 

DISAGREE 

005  MR10705E  (05 

) 00010 

STRONGLY  DISAGREE 

006  M810705M  (M 

) 00001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SUBJ0045 

DESCRIPTION: 

AGREE/DISAGREE:  WOULD  NOT  STUDY  MORE  MATH 

GRADES/ASSESSMENTS ■ 

N08,  S08 , N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

MR  10706 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  MR10706A  (01 

) 00000 

STRONGLY  AGREE 

002  M810706B  (02 

} 10000 

AGREE 

003  M810706C  (03 

) 01000 

UNDECIDED 

004  MR10706D  (04 

) 00100 

DISAGREE 

005  M810706E  (05 

) 00010 

STRONGLY  DISAGREE 

006  M810706M  (M 

} 00001 

MISSING 

CONDITIONING  VARIABLE  ID 

SUBJ004  6 

DESCRIPTION: 

AGREE/ DISAGREE:  ALL 

CAN 

CO  WELL  IN  MATH  IF  TRY 

GRADES /ASSESSMENTS : 

N08 , SOfi,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

M810711 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  MR  107 1 l A (01 

) OOOCO 

STRONGLY  AGREE 

002  MR  1 07 1 1 B (02 

} 10000 

AGREE 

00  3 MR  107 11C  (03 

) 01000 

UNDECIDED 

004  MH 1 07 1 ID  (04 

) 00100 

DISAGREE 

005  MR10711E  (05 

) 00010 

STRONGLY  DISAGREE 

006  MR  1 07 1 1 M <M 

) 00001 

MISSING 

6 

5 


6 

5 


6 

5 


6 

5 


6 

5 


310 


4 

j 


CONDITIONING  VARIABLE  ID:  SCHL0001 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHOOL  LEVEL  AVERAGE  MATH  NORMIT  (MISSING  VS  NON-MISSING) 

N04,  S04 , NOB,  SOB,  N12 
SCH  NORM 

SCHNORM  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  SCHNRM-?  (M 

002  SCHNRM-Y  (8 


SCHOOL  LEVEL  AVERAGE  MATH  NORMIT  MISSING 
SCHOOL  LEVEL  AVERAGE  MATH  NORMIT  HOT -MIS SING 


CONDITIONING  VARIABLE  ID:  SCHL0002 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHOOL  LEVEL  AVERAGE  MATH  NORMIT 

N04,  S04 , N08,  SOB,  N12 

SNRM-LIN 

SCHNORM 

SCALE 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  SNRM-LIN  {# 

002  SNRM-LIN  (M 


) (F8.4) 

) 0 


SCHOOL  LEVEL  AVERAGE  MATH  NORMIT  MEAN 
SCHOOL  LEVEL  AVERAGE  MATH  NORMIT  MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0003 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BEST  DESCRIBES  HOW  4TH  GR  ARE  ORGANIZED? 
N04 , S04 


C030901 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C030901A  (01 

002  C03090LB  (02 

003  C030901C  (03 

004  C030901M  (M 


SELF  CONTAINED 
DEPARTMENTAL I ZED 
REGROUPED 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0004 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


4TH  GRADERS  ASSIGNED  BY  ABILITY /ACHIEVEMENT  LEVEL? 
N04,  S04 


C037101 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037101A  (01 

002  C037 101B  (02 

003  C037101C  (03 

004  C037101N  (04 

005  C037101M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


YES,  BY  MATH  ABILITY 
YES,  READING  ABILITY 
YES,  GENERAL  ABILITY 
NO 

MISSING 


CONDITIONING  VARIABLE  ID;  SCHL0005 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  IS  4TH-GRADER  INSTRUCTED  IN  MATH? 
N04 , S04 


C031212 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C031212A  (01 

002  C031212B  (02 

003  C031212C  (03 

004  C031212D  (04 

005  C031212E  (05 

006  C031212M  (M 


) 00000 
) 10000 
) 01000 
) 00100 
) 00010 
) 00001 


EVERY  DAY 

THREE  OR  FOUR/WEEK 
ONCE  OR  TWICE  A WEEK 
LESS  THAN  ONCE/WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0006 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL; 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  IS  4TH-GRADER  INSTRUCTED  IN  SCIENCE' 
N04 , S04 


C031205 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C031205A  (01 

002  C031205B  (02 

003  C031205C  (03 

004  C031205D  (04 

005  C031205E  (05 

006  C031205M  (M 


> 00000 
) 10000 
) 01000 
) 00100 
) 00010 
) 00001 


EVERY  DAY 

THREE  OR  FOUR /WEEK 
ONCE  OR  TWICE  A WEEK 
LESS  THAN  ONCE/WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0007 

DESCRIPTION:  HOW  OFTEN  IS  4TH-GRADER  INSTRUCTED  IN  READING? 

GRADES /ASSESSMENTS:  N04 , S04 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C031213  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST:  CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


BEST  COPY  AVAILABLE 


31) 


001  C031213A  (01  } 00000 

002  C031213B  (02  ) 10000 

003  C031213C  (03  ) 01000 

004  C031213D  (04  ) 00100 

005  C031213E  (05  ) 00010 

006  C031213M  (M  ) 00001 


EVERY  DAY 

THREE  OR  FOUR/WEEK 
ONCE  OR  TWICE  A WEEK 
LESS  THAN  ONCE/WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING 

VARIABLE  ID: 

SCHL0008 

DESCRIPTION: 

HOW  OFTEN  IS  4 TH -GRADER 

GRADES /ASSESSMENTS : 

N04 , S04 

CONDITIONING 

VAR  LABEL: 

NAEP  ID: 

C031214 

TYPE  OF  CONTRAST: 

CLASS 

001  C031214A 

(01 

) 00000 

002  C031214B 

(02 

) 10000 

003  C031214C 

(03 

) 01000 

004  C031214D 

(04 

) 00100 

005  C031214E 

(05 

) 00010 

006  C031214M 

(M 

) 00001 

INSTRUCTED  IN  ARTS? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  6 

NUMBER  OF  INDEPENDENT  CONTRASTS:  5 

EVERY  DAY 

THREE  OR  FOUR/WEEK 
ONCE  OR  TWICE  A WEEK 
LESS  THAN  ONCE/WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0009 

HAS  MATH  BEEN  IDENTIFIED  AS  A PRIORITY? 

NC4 , 504,  N08,  S08 

C031603  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


3 

2 


001  C031603Y  (01  ) 00 

002  C031603N  (02  ) 10 

003  C031603M  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0010 

HAS  SCIENCE  BEEN  IDENTIFIED  AS  A PRIORITY? 

N04 , S04 , NO 8 , S08 

C031607  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


3 


001  C031607Y  (01  ) 00 

002  C031607N  (02  ) 10 

003  C031607M  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHLOO 1 1 

HAS  READING  BEEN  IDENTIFIED  AS  A PRIORITY? 

N04 , SO 4 

C031601  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


3 

2 


001  CO 3 3 601 Y (01  ) 00 

002  C031601N  (02  ) 10 

003  C031601M  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0012 

HAS  ARTS  BEEN  IDENTIFIED  AS  A PRIORITY7 
N04 , S04,  N08 , S08 

C031610  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


3 

2 


001  C031610Y  (01  ) 00 

002  C031610N  (02  ) 10 

003  C031610M  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 

SCHL0013 

DESCRIPTION: 

HAS  SUBJECT  INTEGRATION  BEEN  A 

PRIORITY? 

GRADES/ ASSESSMENTS : 

N04 , S04 , NOB,  S08 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

C031606 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTPAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C031606Y  (01 

) 00 

YES 

002  C031606N  (02 

) 10 

NO 

003  C031606M  (M 

) 01 

MISSING 

3 


CONDITIONING  VARIABLE  ID. 
DESCRIPTION: 

GRADES /ASSESSMENTS : 


SCHLOO 1 4 

COMPUTERS  AVAILABLE  ALL  THE  TTME  IN  CLASSROOM7 
N04 , S04 , NOR,  S08,  N12 


r ^ 
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CONDITIONING  VAR  LABEL: 

NAEP  ID:  C035701 

TYPE  OF  CONTRAST:  CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

NUMBER  OF  INDEPENDENT  CONTRASTS:  2 


001  C035701Y  (01  ) 00 

002  C035701N  (02  ) 10 

003  C03570IM  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0015 

COMPUTERS  GROUPED  IN  SEPARATE  LAB  AND  AVAILABLE? 

N04,  S04 , N08 , S08,  N12 

C035702  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


3 

2 


001  C035702Y  (01  ) 00 

002  C035702N  (02  ) 10 

003  C035702M  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL001 6 

COMPUTERS  AVAILABLE  TO  BRING  TO  ROOM  WHEN  NEEDED? 

N04,  S04,  NOB , SOB,  N12 

CO 3 5703  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


3 

2 


001  C035703Y  (01  ) 00 

002  C035703N  (02  ) 10 

003  C035703M  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0017 

DESCRIPTION:  SCHOOL  W/  SPECIAL  FOCUS  ON  MATH? 

GRADES /ASSESSMENTS:  N04,  S04 , N08,  SOS,  N12 

CONDITIONING  VAR  LABEL: 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


NAEP  ID:  C037201 

TYPE  OF  CONTRAST:  CLASS 


2 

1 


001  C037201Y  (01  ) 0 

002  C037201M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0018 

SCHOOL  W / SPECIAL  FOCUS  ON  SCIENCE? 

N04 , S04 , NO 8 , SOB,  N12 

C037202  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C037202Y  (01  ) 0 

002  C037202M  (M  1 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0019 

DESCRIPTION:  SCHOOL  W/  SPECIAL  FOCUS  ON  MATH? 

GRADES /ASSESSMENTS:  N04,  S04 

CONDITIONING  VAR  LABEL: 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


NAEP  ID:  C037207 

TYPE  OF  CONTRAST:  CLASS 


001  C037207Y  (01  ) 0 

002  C037207M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0020 

DESCRIPTION:  SCHOOL  W/  SPECIAL  FOCUS  ON  ARTS? 

GRADES/ASSESSMENTS:  N04 . S04 , N08,  S08,  N12 

CONDITIONING  VAR  LABEL: 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


NAEP  ID:  C037204 

TYPE  OF  CONTRAST:  CLASS 


001  C037204Y  (01  ) 0 

002  C037204M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0021 

DESCRIPTION:  SCHOOL  W/  SPECIAL  FOCUS  ON  OTHER? 

GRADES /ASSESSMENTS:  N04,  S04,  N08,  S08 , N12 

CONDITIONING  VAR  LABEL: 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS; 


NAEP  ID:  C03720S 

TYPE  OF  CONTRAST:  CLASS 


2 

1 


001  C03720SY  (01  ) 0 

002  C037205M  (M  ) 1 


YES 

MISSING 


best  copy  available 

051 
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CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST; 


SCHL0022 

SCHOOL  NOT  A SPECIAL  FOCUS  SCHOOL? 

N04 , S04 , NOB , S08 , N12 

C037206  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C037206Y  (01  ) 0 

002  C037206M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0023 

SCHOOL  FOLLOW  DISTRICT/ STATE  MATH  CURRICULUM? 

N04 , S04,  N08 , S08,  N12 

C037301  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037301Y  (01  ) 0 

002  C037301M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL; 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHLC024 

SCHOOL  FOLLOW  DISTRICT/ STATE  SCIENCE  CURRICULUM? 

N04 , S04,  N08 , S08,  N12 

C037302  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C037302Y  (01  ) 0 

002  CC37302M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  Or  CONTRAST: 


SCHL0025 

SCHOOL  FOLLOW  DISTRICT/S  i'ATE  READING  CURRICULUM? 

N04 , S04 

C037303  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C037303Y  (01  ) 0 

0C2  C03?303M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0026 

SCHOOL  FOLLOW  DISTRICT/ STATE  ARTS  CURRICULUM? 

N04,  S04 , N08,  S08,  N12 

C037304  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C037304Y  (01  ) 0 

002  C0373C4M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID. 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0027 

SCHOOL  FOLLOW  DISTRICT /STATE  FOR  NONE  OF  ABOVE9 
NO 4 , S04 , NOS,  S08 , N12 

C037305  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037305Y  (01  ) 0 

002  C037J05M  (M  I 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 

SCHL0028 

DESCRIPTION: 

SCHOOL  SPONSER  4TH  CRDS 

FIELD  TRIP  FOR  MATH? 

GRADES /ASSESSMENTS : 

N04 , S04 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

C037401 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C037401Y  (Cl 

) 0 

YES 

002  C037401M  (M 

) 1 

MISSING 

2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL. 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0029 

SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  SCIENCE? 

N04,  S04 

C037402  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS • 


001  CO37402Y  (01  ) 0 

002  C037402M  (M  » 1 


YES 

MISSING 


314 


t*  t-  'A 


CONDITIONING  VARIABLE  ID:  SCHL0030 

DESCRIPTION:  SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  READING? 

GRADES/ ASSESSMENTS:  N04 , S04 

CONDITIONING  VAR  LABEL: 

NAEP  ID-  CO 3 74 03  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST:  CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


id  C037403Y  (01 
'32  CC-37403M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0031 


DESCRIPTION: 

GRADES  ASSESSMENTS: 
CONDITIONING  VAR  LA3EL ■ 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  ARTS’ 
MO  4 . S04 


C0374C4 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


C 0 1 CO 3 74 04 Y {Cl 
002  C03n404M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LAB El 
NAEP  ID: 

TYPE  CF  CONTRAST: 


SCHL0032 

SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  OTHER? 
N04 , S04 


C037405 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTPASTS : 


001  CC3',40r>Y  {01 

002  C0374C5M  IM 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0C3 3 

DESCRIPTION:  SCHOOL  SPONSER  4TH  GRDS  FIELD  TRIP  FOR  NONE  ABOVE? 

GRADES, ASSESSMENTS:  N04 , S04 

CONDITIONING  VAR  LABEL:  „ „ 

NAEP  1D.  C037406  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST:  CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  CC374C6Y  (01 

002  CO  3n406M  iM 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0034 


DESCRIPTION: 

GRADES ' ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  CF  CONTRAST: 


4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  MATH7 
N04 , S04 


C037501 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  CF  INDEPENDENT  CONTRASTS: 


CGI  CC375C1Y  ‘01 
032  CO 3 "50 IM  <M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0035 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABE! 
NAEP  ID: 

TYPE  OF  CONTRAST: 


4TH  GRADERS  IN  EXTRACURR  ACTS  FOP.  SCIENCE7 
NO 4 , SO 4 


C03750: 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS. 


001  CC 3 7 502Y  (01 

002  C 0 3 " 5 0 2 H (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0G36 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  READING' 
N04 , SOI 


C037503 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRAST! 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


0C1  C0375Q3Y  (01 
0C2  C03"e'O3M  (M 


YES 

MISS INC 


CONDITIONING  VARIABLE  ID:  SCHL0037 


DESCRIPTION: 

GRADES  ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
KAEP  ID: 

TYPE  OF  CONTRAST: 


4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  ARTS7 
N04,  S04 


roi 7 so 4 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


r-01  COV/StMY  (01 


BEST  COPY  AVAILABLE 


;*  1 f*  r') 


002  C037504M  (M 


) 


1 


MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL003  8 

4TH  GRADERS  IN  EXTRACURR  ACTS  FOR  NONE  OF  ABOVE? 

N04 , S04 

C037505  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037505Y  (01  ) 0 

002  C03750SM  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID; 

SCHL0039 

DESCRIPTION: 

4TH  GRADERS 

IN  SUMMER  PROGRAMS  IN  MATH? 

GRADES /ASSESSMENTS : 

N04 , S04 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

C037601 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C037601Y  (01  ] 

1 0 

YES 

002  C037601M  (M  ] 

1 1 

MISSING 

2 

1 


CONDITIONING  VARIABLE  ID:  SCHL0040 


DESCRIPTION: 

4TH  GRADERS 

IN  SUMMER  PROGRAMS  IN  SCIENCE? 

GRADES/ASSESSMENTS : 

N04 , S04 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

C037602 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C037602Y  (01 

) o 

YES 

002  C037602M  (M 

> 1 

MISSING 

2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0041 

4TH  GRADERS  IN  SUMMER  PROGRAMS  IN  READING? 

N04 , S04 

C037603  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C037603Y  (01  ) 0 

002  C037603M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 

SCHL0042 

DESCRIPTION: 

4TH  GRADERS 

IN  SUMMER  PROGRAMS  IN  ARTS? 

GRADES /ASSESSMENTS : 

N04 , S04 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

C037604 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C037604Y  (01  1 

1 0 

YES 

002  C037604M  (M 

) 1 

MISSING 

2 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0043 

4TH  GRADERS  IN  SUMMER  PROGRAMS  IN  NONE  OF  ABOVE? 

N04 , S04 

C037605  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C037605Y  (01  ) 0 

002  C03760SM  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


5CHL0044 

V/HICH  BEST  DESCRIBES  PRIMARY  WAY  LIBRARY  STAFFED? 

N04 , S04 , NOB,  S08,  N12 

CO 3 6 601  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


S 

4 


001 

C036601A 

(01 

) 0000 

002 

C036601B 

(02 

) 1000 

00  3 

CO  3 6601C 

(03 

) 0100 

004 

C036601D 

(04 

) 0010 

005 

CO  3 660 1M 

(M 

) 0001 

NO  LIBRARY  IN  SCHOOL 
LIBRARY -NO /VOL  STAFF 
PART-TIME  STAFF 
FULL-TIME  STAFF 
MISSING 


CUNLU'l  I ON  I NO  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 


SCHL0045 

INVOLVE  PARENTS  AS  AIDES  IN  CLASSROOM’ 
N04 , S04.  NOR,  S08,  N12 


316 


i K * 


NAEP  ID: 

TYPE  OF  CONTRAST 


C032207 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032207A  (01 

002  C032207B  (02 
J03  C032207N  (03 
f.04  C032207M  (M 


YES,  ROUTINELY 
YES,  OCCASIONALLY 
NO 

M'""  SING 


CONDITIONING  VARIABLE  ID:  SCHL0046 


DESCRIPTION: 

GI  ADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL : 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HAVE  PARENTS  REVIEW/ SIGN  HOMEWORK? 
N04,  S04,  N08 , S08,  N12 


C032209 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032209A  (01 

002  C032209B  (02 

003  C032209N  (03 

004  C032209M  (M 


YES,  ROUTINELY 
YES,  OCCASIONALLY 
NO 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0047 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ASSIGN  HOMEWORK  STUDENTS  DO  WITH  PARENTS? 
N04,  S04 , N08,  S08,  N12 


C032210 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032210A  (01 

002  C032210R  (02 

003  C032210N  '03 

004  C032210M  (M 


YES,  ROUTINELY 
YES,  OCCASIONALLY 
NO 

MISSING 


CONDITIONING  VARIABLE  £D:  SCHL0048 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HAVE  A PARENT  VOLUNTEER  PROGRAM? 
N04,  S04,  NOS,  S08,  N12 


C0322U 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032211A  (01 

002  C032211B  (02 

003  C032211N  (03 

004  C032211M  (M 


YES,  ROUTINELY 
YES,  OCCASIONALLY 
NO 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0G49 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL; 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  C037701A  (01 

002  C037701B  (02 

003  C037701C  (03 

004  C037701D  (04 

005  C037701M  (M 


WHAT  % OF  PARENTS  IN  PARENT-TEACHER  ORGS? 
N04,  S04 , N08,  SO0.  N12 


C037701 

CLASS 

) 0000 
) 1000 
) 0100 
} 0010 
) 0001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

0-25% 

26-50% 

51-75% 

76-100% 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0050 

WHAT  % OF  PARENTS  IN  OPEN  HOUSE/BACK  SCHOOL  NIGHT? 
N04 , S04 , NOB,  SOB,  N12 


C037702 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037702A  (01 

002  C037702E  (02 

003  C037702C  (03 

004  C037702D  (04 

005  C037702M  (M 


) 0000 
) 1000 
} 0100 
) 0010 
) 0001 


0-25% 

26-50% 

51-75% 

76-100% 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0051 

DESCRIPTION:  WHAT  % OF  PARENTS  IN  PARENT- TEACH EP  CONFERENCES? 

GRADES /ASSESSMENTS:  N04 , S04 , N08,  S08,  N12 

CONDITIONING  VAR  L/.BEL  ■ 

NAEP  ID:  C037703  TOTAL  NUMBER  OF 

TYPE  OF  CONTRAST:  CLASS  NUMBER  OF  INDEF 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037703A  (01 

002  C037703B  (02 

003  C037703C  (03 

004  C037703D  (04 


1 0000 
) 1000 
) 0100 
) 0010 


0-25% 

26-50% 

51-75% 

■76-100% 


T > T' 

o o o 


best  copy  available 


317 


005  C037703M  (M 


) 0001 


MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0052 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


WHAT  % PARENTS  INVOLVED  MAKING  CURRICULUM  DECISION 
N04 , S04 , N08 , SOB,  N12 


C037704 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037704A  (01 

002  CO37704B  (02 

003  C037704C  (03 

004  C037704D  (04 

005  C037704M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


0-25% 

26-50% 

51-75% 

76-100% 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0053 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


WHAT  % OF  PARENTS  IN  VOLUNTEER  PROGRAMS? 
N04 , S04 , NOB , S08,  N12 


C037705 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037705A  (01 

002  CO37705B  (02 

003  C037705C  (03 

004  C037705D  (04 

005  C037705M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


0-25% 

26-50% 

51-75% 

76-100% 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


IS  STUDENT  ABSENTEEISM  A PROBLEM  IN  YOUR  SCHOOL? 
N04 , S04 , N08 , S08,  N12 


C032402 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032402A  (01 

002  C032402B  (02 

003  C032402C  (03 

004  C032402D  (04 

005  C032402M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0055 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


IS  STUDENT  TARDINESS  A PROBLEM  IN  YOUR  SCHOOL? 
N04 , S04 , N08 , S08,  N12 


C032401 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032401A  (01 

002  C032401B  (02 

003  C032401C  (03 

004  C032401D  (04 

005  C032401M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0056 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ARE  PHYSICAL  CONFLICTS  A PROBLEM  IN  YOUR  SCHOOL? 
N04 , S04 , NOB,  SOB,  N12 


C032404 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032404A  (01 

002  C032404B  (02 

003  C032404C  (03 

004  C032404D  (04 

005  C032404M  (M 


) 000'.’ 
) 1000 
) 0100 
) 0010 
) 0001 


SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0057 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL; 
NAEP  ID: 

TYPE  OF  CONTRAST: 


IS  TEACHER  ABSENTEEISM  A PROBLEM  IN  YOUR  SCHOOL? 
N04 , S04 , N08 , SOB,  N12 


C032406 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032406A  (01 

002  C032406B  (02 

003  C032406C  (03 

004  C032406D  (04 

005  C032406M  (M 


) 0000 
) 1000 
) 0100 
> 0010 
) 0001 


SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


O r;  n 


CONDITIONING  VARIABLE  ID:  SCHL0058 

DESCRIPTION:  ARE  RACE/CULT.  CONFLICTS 

GRADES /ASSESSMENTS:  N04,  S04 , N08,  S08,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C032407 

TYPE  OF  CONTRAST:  CLASS 

001  C032407A  (01  ) 0000 

002  C032407B  (02  ) 1000 

003  C032407C  (03  ) 0100 

004  C032407D  (04  ) 0010 

005  C032407M  (M  ) 0001 


A PROBLEM  IN  YOUR  SCHOOL? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  5 

NUMBER  OF  INDEPENDENT  CONTRASTS:  4 

SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 

/ 


CONDITIONING  VARIABLE  ID:  SCHL0059 

DESCRIPTION:  IS  STUDENT  HEALTH  A 

GRADES /ASSESSMENTS:  N04,  S04,  N08,  S08, 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C032408 

TYPE  OF  CONTRAST:  CLASS 


PROBLEM  IN  YOUR  SCHOOL? 

N12 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032408A  (01  ) 0000 

002  C032408B  (02  ) 1000 

003  C032408C  (03  ) 0100 

004  C032408D  (04  ) 0010 

005  C032408M  (M  > 0001 


SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


5 

4 


CONDITIONING  VARIABLE  ID:  SCHL0060 

DESCRIPTION:  IS  LACK  OF  PARENT  INVLMT 

GRADES /ASSESSMENTS:  N04 . S04,  NOB,  S08,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C032409 

TYPE  OF  CONTRAST:  CLASS 

001  C032409A  (01  ) 0000 

002  C032409B  (02  ) 1000 

003  C032409C  (03  ) 0100 

004  C032409D  (04  ) 0010 

005  C032409M  (M  ) 0001 


A PROBLEM  IN  YOUR  SCHOOL? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  5 

NUMBER  OF  INDEPENDENT  CONTRASTS:  4 

SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0061 

DESCRIPTION:  IS  STUD  USE  OF  ALCOHOL  A 

GRADES/ASSESSMENTS:  N04 , S04,  N08,  SOB,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C032410 

TYPE  OF  CONTRAST:  CLASS 

001  C032410A  (01  ) 0000 

002  C032410B  (02  ) 1000 

003  C032410C  (03  ) 0100 

004  C032410D  (04  ) 0010 

005  C032410M  (M  ) 0001 


PROBLEM  IN  YOUR  SCHOOL? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  5 

NUMBER  OF  INDEPENDENT  CONTRASTS:  4 

SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0062 

DESCRIPTION:  IS  STUDENT  TOBACCO  USE  A 

GRADES /ASSESSMENTS:  N04 , S04,  N08,  S08,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C032411 

TYPE  OF  CONTRAST:  CLASS 

001  C032411A  (01  ) 0000 

002  C032411B  (02  ) 1000 

003  C032411C  (03  ) 0100 

004  C032411D  (04  ) 0010 

005  C032411M  (M  ) 0001 


PROBLEM  IN  YOUR  SCHOOL? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  5 

NUMBER  OF  INDEPENDENT  CONTRASTS:  4 

SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0063 

DESCRIPTION:  IS  STUDENT  DRUG  USE 

GRADES /ASSESSMENTS : N04,  S04 , NOB,  S08, 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C032412 

TYPE  OF  CONTRAST:  CLASS 

001  C032412A  (01  ) 0000 

002  C032412B  (02  ) 1000 

003  C032412C  (03  ) 0J00 

004  C032412D  (04  ) 0030 

005  C032432M  <M  3 0001 


A PROBLEM  IN  YOUR  SCHOOL? 

N12 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS; 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


5 

4 


CONDITIONING  VARIABLE  ID: 
DESCRI  iTICN : 

GRADES /ASSESSMENTS : 


SCHL0064 

ARE  GANG  ACTIVITIES  A PROBLEM  IN  YOUR  SCHOOL? 
N04,  S04 , NOR,  S08,  N12 


BEST  copy  available 


319 


357 


CONDITIONING  VAR  LABEL- 

NAEP  ID:  C032413 

TYPE  OF  CONTRAST:  CLASS 

001  C032413A  (01  ) 0000 

002  C032413B  (02  J 1000 

003  C03241  3C  < C 3 ) 0100 

004  C032 4 i 3D  (04  ) 0010 

005  C032413M  (M  ) 0001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  C032414A  (01 

002  C032414B  (02 

003  C032414C  (03 

004  C032414D  (04 

005  C032414M  (M 


IS  STUDENT  MISBEHAVIOR  A PROBLEM  IN  YOUR  SCHOOL? 
NO 4 , S04 , N08 , S08,  N12 


C032414 

CLASS 

» 0000 
) 1000 
) 0100 
) 0010 
) 0001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0066 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


IS  STUDENT  CHEATING  A PROBLEM  IN  YOUR  SCHOOL? 
N04,  S04 , N08 , SOS,  N12 


C032415 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032415A  (01 

002  C032415B  (02 

003  C032415C  (03 

004  C032415D  (04 

005  C032415M  (K 


SERIOUS 

MODERATE 

MINOR 

NOT  A PROBLEM 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0067 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

I 'PE  OF  CONTRAST : 


IS  TEACHER  MORALE  POS . OR  NEG . ? 
N04 , S04 , N08 , S08,  N12 


C032502 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032502A  (01 

002  C032502B  (02 

003  C032502C  (03 

004  C032502D  (04 
COS  C0325C2M  <M 


VERY  POSITIVE 
SOMEWHAT  POSITIVE 
SOMEWHAT  NEGATIVE 
VERY  NEGATIVE 
MISSING 


CONDITIONING  VARIABLE  ID-  SCHL0068 


DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LAEEL : 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ARE  STUDENT  ATTITUDES  TO  ACADEMICS  POS.  OR  NEG.? 
N04 , S04 , N08 , S08.  N12 


C032503 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032503A  Cl 

002  CG32503B  < 12 

003  CO 3 2 50 JC  (C-i 

004  CO 3 2 50 3D  (04 

005  CO 3 2502 M (M 


VERY  POSITIVE 
SOMEWHAT  POSITIVE 
SOMEWHAT  NEGATIVE 
VERY  NEGATIVE 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0069 


DESCRIPTION: 

GRADES  ASSESSMENTS: 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


IS  PARENT  SUPPORT  FOR  ACHIEVEMENT  POS.  OR  NEG.? 
NO 4 , S04 , NOS , S08 , N12 


C032505 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C032505A  (Cl 

002  C032505B  (02 
C03  C 032505C  (03 
CC4  C0325C5D  (04 
005  C0325G5M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


VERY  POSITIVE 
SOMEWHAT  POSITIVE 
SOMEWHAT  NEGATIVE 
VERY  NEGATIVE 
MISSING 


CONDITIONING  VARIABLE  ID:  SCHL007G 


DESCRIPTION: 

GRAFFS ASSESSMENTS: 
CONDITIONING  YAP  LABEL 
NAET  It . 

TYFE  ?Y  C'NTFAST- 


IS  REGARD  FOR  SCHOOL  PROPERTY  POS.  OR  NEG.? 
N04,  S04 , NOB , SOB,  N12 


CO  32  506 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CON TRASTS : 


c r v 


001  C032506A  (01 

002  C0325Q6B  (02 

003  C032506C  (03 

004  C032506D  (04 

005  C032506M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


VERY  POSITIVE 
SOMEWHAT  POSITIVE 
SOMEWHAT  NEGATIVE 
VERY  NEGATIVE 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0071 

% ABSENT  ON  AVERAGE  DAY? 
N04,  S04,  N08 , S08 , N12 

C0336C1 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C033601A  (01  ) 

0000 

0-2% 

002  C033601B  (02  ) 

1000 

3-5% 

003  C033601C  (03  ) 

0100 

6-10% 

004  C033601D  (04  ) 

0010 

MORE  THAN  10% 

005  C033601M  (M  ) 

0001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCHL0072 

DESCRIPTION: 

WHAT  % OF 

TEACHERS  ABSENT  ON  GIVEN  DAY? 

GRADES /ASSESSMENTS : 

N04 , S04 , 

N08 , SOB,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C036501 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C036501A  (01  ) 

0000 

0-2% 

002  C036501B  (02  ) 

1000 

3-5% 

003  C036501C  (03  5 

0100 

6-10% 

004  C036501D  (04  ) 

0010 

MORE  THAN  10% 

005  C036501K  (M  ) 

0001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCHL0073 

DESCRIPTION: 

% OF  STUDS  EROLLED  AT  START 

OF  YR  EROLLED  AT  END? 

GRADES /ASSESSMENTS : 

N04,  S04 , 

NOB,  S08 , N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C037801 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C037801A  (01 

00000000 

98-100% 

002  C0378C1B  (02 

10000000 

95-97% 

003  C037801C  (03 

01000000 

90-94% 

004  C037801D  (04 

00100000 

80-89% 

005  C037801E  (05 

00010000 

70-79% 

006  C037801F  (06 

00001000 

60-69% 

007  C037801G  (07 

00000100 

50-59% 

008  C037S01H  (08 

00000010 

LESS  THAN  50% 

009  C037801M  (M 

00000001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCKL007  4 

DESCRIPTION: 

% OF  4TH 

GRADERS  HELD  BACK 

St  REPEATING  4TH  GRADE? 

GRADES/ASSESSMENTS : 

N04 , S04 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C037901 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C037901A  (01 

) 00000 

0% 

002  C037901B  (02 

) 10000 

1-2% 

003  C037901C  (03 

) 01000 

3-5% 

004  C037901D  (04 

) 00100 

6-10% 

005  C037901E  (05 

) 00010 

MORE  THAN  10% 

006  C0379O1M  < M 

) 00001 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCHL0075 

DESCRIPTION; 

% OF  FULL  TIME  TEACHERS  LEFT  BEFORE  END  OF  YR? 

GRADES /ASSESSMENTS : 

N04 , S04 

. NOB,  SOS,  N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C038001 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C038001A  (01 

) 0000000 

0% 

002  C038001B  (02 

) 1000000 

1-2% 

003  C038001C  (03 

) oiooooo 

3-5% 

004  C038001D  (04 

} 0010000 

6-10% 

005  C038001E  (05 

) oooiooo 

11-15% 

006  CO 3 8001 F (06 

) 0000100 

16-20% 

007  C038001G  (07 

) 0000010 

MORE  THAN  20% 

OOH  C038001M  (M 

) 00000C1 

MISSING 

5 

4 


5 

4 


9 

8 


6 

5 


8 

7 


CONDITIONING  VARIABLE  ID:  SCHL0076 
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BEST  COPY  AVAILABLE 


DESCRIPTION: 

IS  SCHOOL 

IN  NATIONAL  SCHOOL 

LUNCH  PROGRAM? 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 

N04 , S04 , 

N08 , S08 , N12 

NAEP  ID: 

C038301 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C038301Y  (01 

> 00 

YES 

002  C038301N  (02 

) 10 

NO 

003  C038301M  (M 

) 01 

KISSING 

CONDITIONING  VARIABLE  ID: 

SCHL0077 

DESCRIPTION: 

SCHOOL  RECEIVE 

CHAP 

1/TITLE  1 

FUNDING? 

GRADES /ASSESSMENTS : 

NO 4 , S04,  N08 , 

S08 , 

N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C038801 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  C038801Y  (01  ) 

00 

YES 

002  C038801N  (02  ) 

10 

NO 

003  C038801M  (M  ) 

01 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

SCHL0078 

DID  PRINCIPAL  FILL 

OUT  THIS 

QUESTIONNAIRE 

GRADES /ASSESSMENTS : 

N04 , S04 , N08 , S08, 

N12 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

C034101 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS : 

001  C034101Y  (01  ; 

1 0 

YES 

002  C034101K  (M 

1 1 

MISSING 

3 

2 


3 

2 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0079 

DID  HEADMASTER/HEADMISTRESS  FILL  OUT  QUESTIONNAIRE 
N04,  S04 , N08 , S08 , N12 

C034102  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C034102Y  (01  ) 0 

002  C034102M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0080 

DID  HEAD  TEACHER  FILL  OUT  THIS  QUESTIONNAIRE 
N04 , S04 , N08 , S08 , N12 

C034103  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS : 


2 

1 


001  C034103Y  (01  ) 0 

002  C034103M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL008 1 

DID  VICE  PRINCIPAL  FILL  OUT  THIS  QUESTIONNAIRE 
N04,  S04 , N08 , SOB,  N12 

C034104  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C034104Y  (01  ) 0 

002  C034104M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0082 

DID  COUNSELOR  FILL  OUT  THIS  QUESTIONNAIRE 
N04 , S04 , N08.  S08,  N12 

C034105  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C034105Y  (01  ) 0 

002  C034105M  (M  } 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTFAST: 


SCHL0083 

DID  CURRICULUM  COORD  FILL  OUT  THIS  QUESTIONNAIRE 
NO 4 , S04 , N08 , S08,  N12 

C034106  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  C034106Y  (01  ) 0 

002  C034106M  (M  ) 1 


YES 

MISSING 
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CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0084 

DID  TEACHER  FILL  CUT  THIS  QUESTIONNAIRE 
N04,  S04 , N08 , S08,  N12 

C034107  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C034107Y  (01  ) 0 

002  C034107M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0085 

DID  SECRETARY  FILL  OUT  THIS  QUESTIONNAIRE 
N04 , S04  < NC8 , S08,  N12 

C034108  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C034108Y  (01  ) 0 

002  C034108M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0086 

DID  OTHER  PERSON  FILL  OUT  THIS  QUESTIONNAIRE 
N04,  S04,  N08 , S08,  N12 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


C034109 

CLASS 


001  C034109Y  (01  ) 0 

002  C034109M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0087 

BEST  DESCRIBES  HOW  8TH  GRADES  ARE  ORGANIZED? 

N08,  S08 

C034201  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS : 


001  C034201A  (01  ) 000 

002  C034201B  (02  ) 100 

003  C034201C  (03  ) 010 

004  C034201M  (M  ) 001 


SELF-CONTAINED 
SEMI -DEPARTMENTALIZE 
DEPARTMENTALIZED 
MISSING 


4 

3 


CONDITIONING  VARIABLE  ID:  SCHL0088 

DESCRIPTION:  ARE  8 TH -GRADERS  ASSIGNED  TO  MATH  BY  ABILITY? 


GRADES /ASSESSMENTS:  N08,  SOB 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C034402 

TYPE  OF  CONTRAST:  CLASS 

001  C034402Y  (01  ) 00 

002  C034402N  (02  ) 10 

003  C034402M  (M  ) 01 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

NUMBER  OF  INDEPENDENT  CONTRASTS:  2 

YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  5CHL0089 

DESCRIPTION:  ARE  8TH-GRADERS  ASSIGNED  TO  SCIENCE  BY  ABILITY? 


GRADES /ASSESSMENTS:  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  C0344C3 

TYPE  OF  CONTRAS^.  CLASS 

001  C034403 Y (01  ) 00 

002  C034403N  (02  ) 10 

003  C034403M  <M  >01 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

NUMBER  OF  INDEPENDENT  CONTRASTS:  2 

YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHLC090 

DESCRIPTION-  ARE  8TH-GRADERS  ASSIGNED  TO  ENGLISH  BY  ABILITY? 


GRADES- ASSESSMENTS : NO®.  S08 

CONDITIONING  VAP.  LABEL: 

NAEP  ID:  CO 3 44 01 

TYPE  OK  CONTRAST:  CLASS 

001  C034401Y  (01  > 00 

002  CO 3 440 IN  (02  » 10 

003  Cn34401M  (M  ) 01 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

NUMBER  OF  INDEPENDENT  CONTRASTS:  2 

YES 

NO 

MISSING 


CONDITIONING  VARIABLE  IP: 
DESCRIPTION: 

GRADES / ASSESSMENTS : 


SCHL009 1 

ARE  BTH- GRADERS  ASSIGNED  TO  ARTS  BY  ABILITY 

NO a,  soft 
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CONDITIONING  VAR  LABEL: 


NAEP  ID:  C034406 

TYPE  OF  CONTRAST:  CLASS 

001  C034406Y  (01  ) 00 

002  C034406N  (02  ) 10 

003  C034406M  (M  ) 01 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0092 

HOW  OFTEN  8TH  GRDS  RECEIVE  COMP  SCI  INSTRUCTION? 

N08 , S08 

CO 3 4 510  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

CO 3 4 510A 

(01 

) 00000 

002 

C034510B 

(02 

) 10000 

003 

C034510C 

(03 

) 01000 

004 

C034510D 

(04 

) 00100 

005 

C034510E 

(05 

) 00010 

006 

C034510M 

(M 

) 00001 

EVERY  DAY 

3 OR  4 TIMES  A WEEK 
ONCE  OR  TWICE /WEEK 
LESS  THAN  ONCE /WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


001 

C034511A 

(01 

) 00000 

002 

C034511B 

(02 

) 10000 

003 

C034511C 

(03 

) 01000 

004 

C03451 ID 

(04 

) 00100 

005 

C034511E 

(05 

) 00010 

006 

C034511M 

(M 

) 00001 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

EVERY  DAY 

3 OR  4 TIMES  A WEEK 
ONCE  OR  TWICE/WEEK 
LESS  THAN  ONCE /WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


VARIABLE  ID:  SCHL0093 

HOW  OFTEN  8TH  GRDS  RECEIVE  MATH  INSTRUCTION? 
NOS,  SOB 

C034511 
CLASS 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0094 

HOW  OFTEN  8TH  GRDS  RECEIVE  SCIENCE  INSTRUCTION? 

N08 , S08 

C034512  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

C034512A 

(01 

) 00000 

002 

C034512B 

(02 

) 10000 

003 

C034512C 

(03 

) 01000 

004 

C034512D 

(04 

) 00100 

005 

C034512E 

(05 

) 00010 

006 

C034512M 

(M 

) 00001 

EVERY  DAY 

3 OR  4 TIMES  A WEEK 
ONCE  OR  TWICE/WEEK 
LESS  THAN  ONCE /WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0095 

HOW  OFTEN  8TH  GRDS  RECEIVE  ENGLISH  INSTRUCTION? 

NO 8 , S08 

C034513  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

C034513A 

(01 

) 00000 

002 

C034513B 

(02 

) 10000 

003 

C034513C 

(03 

) 01000 

004 

C034513D 

(04 

) 00100 

005 

C034513E 

(05 

) 00010 

006 

C034513M 

(M 

) 00001 

EVERY  DAY 

3 OR  4 TIMES  A WEEK 
ONCE  OR  TWICE /WEEK 
LESS  THAN  ONCE/WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0096 

HOW  OFTEN  8TH  GRDS  RECEIVE  ARTS  INSTRUCTION? 

N08 , S08 

C034514  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001 

C034514A 

(01 

1 00000 

002 

C034514B 

(02 

) 10000 

003 

C034514C 

(03 

) 01000 

004 

C034514D 

(04 

) 00100 

005 

C034514E 

(05 

) 00010 

006 

C034514M 

(M 

) 00001 

EVERY  DAY 

3 OR  4 TIMES  A WEEK 
ONCE  OR  TWICE/WEEK 
LESS  THAN  ONCE /WEEK 
SUBJECT  NOT  TAUGHT 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 


5CHL0097 

HAS  ENGLISH  BEEN  IDENTIFIED  AS  A PRIORITY? 
N08 , S08 


3 

2 


6 

5 


6 

5 


6 

5 


6 

5 


6 

5 


324 


CONDITIONING  VAR  LABEL: 


NAEP  ID:  C031611 

TYPE  OF  CONTRAST:  CLASS 

001  C031611Y  (01  ) 00 

002  C031611N  (02  ) 10 

003  C031611M  (M  ) 01 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0098 

SCHOOL  OFFER  BTH  GR  STUDS  ALGEBRA  FOR  HS  CREDIT? 

N08 , SO 8 

C034601  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C034601Y  (01  ) 00 

002  C034601N  (02  ) 10 

003  C034601M  (M  ) 01 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0099 

SCHOOL  W/  SPECIAL  FOCUS  ON  ENGLISH? 

N08 , S08 , N12 

C037203  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037203Y  (01  ) 0 

002  C037203M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0100 

SCHOOL  FOLLOW  DISTRICT/ STATE  ENGLISH  CURRICULUM? 

N08,  SOS , N12 

C037306  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C037306Y  (01  ) 0 

002  C037306M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0101 

SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  MATH? 

N08 , S08 

C039401  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039401Y  (01  ) 0 

002  C039401M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0102 

SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  SCIENCE? 

N08,  S08 

C039402  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039402Y  (01  ) 0 

002  C039402M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL01 03 

SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  READING? 

N08,  S08 

C039403  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039403Y  (01  ) 0 

002  C039403M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHI.0104 

SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  ARTS? 

N08 , S08 

C039404  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039404Y  (01  ) 0 

002  C039404M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0105 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  OTHER? 
N08,  S08 


C039405 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039405Y  (01 

002  C039405M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0106 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  CF  CONTRAST: 


SCHOOL  SPONSER  8TH  GRDS  FIELD  TRIP  FOR  NONE  ABOVE? 
NOB , S08 


C039406 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039406Y  (01 

002  C039406M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0107 

8TH  GRADERS  IN  EXTRACTOR  ACTS  FOR  MATH? 
NOS,  S08 


C039501 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039501Y  (01 

002  C039501M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0108 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAk  J ABEL : 
NAEP  ID: 

TYPE  OF  CONTRAST: 


8TH  GRADERS  IN  EXTRACTOR  ACTS  FOR  SCIENCE? 
N08 , S08 


C039502 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039502Y  (01 

002  C039502M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0109 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


8TH  GRADERS  IN  EXTRACTOR  ACTS  FOR  ENG /LANG  ARTS? 
N08 , S08 


C039503 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039503Y  (01 

002  C039503M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0110 


DESCRIPTION; 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


8TH  GRADERS  IN  EXTRACURR  ACTS  FOR  ARTS? 
N08 , SOS 


C039504 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039504Y  (01 

002  C039504M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0111 


DESCRIPTION: 

GRADES/ ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID; 

TYPE  OF  CONTRAST: 


8TH  GRADERS  IN  EXTRACURR  ACTS  FOR  NONE  OF  ABOVE? 
N08 , SOS 


C039505 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  C039505Y  (01 

002  C039505M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  SCHL0U2 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


8TH  GRADERS  IN  SUMMER  PROGRAMS  IN  MATH? 
N08 , SOB 


C039601 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS; 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  CO  1900.’ Y (01 

002  CO 3 960 1M  (M 


YES 

MISSING 
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CONDITIONING  VARIABLE  ID:  SCHL0113 


DESCRIPTION: 

8TH  GRADERS 

IN 

SUMMER 

PROGRAMS 

IN  SCIENCE? 

GRADES /ASSESSMENTS : 

N08 , S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C039602 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

2 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

1 

001  C039602Y  (01  I 

1 0 

YES 

002  C039602M  (M 

) 1 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCHL0114 

DESCRIPTION: 

8TH  GRADERS 

IN 

SUMMER 

PROGRAMS 

IN  ENG /LANG  ARTS? 

GRADES /ASSESSMENTS : 

NO 8 , SOB 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C039603 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

2 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

1 

001  C039603Y  (01 

) o 

YES 

002  C039603M  <M 

) 1 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCHL0115 

DESCRIPTION: 

8TH  GRADERS 

IN 

SUMMER 

PROGRAMS 

IN  ARTS? 

GRADES /ASSESSMENTS : 

NOS,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C039604 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

2 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

1 

001  C039604Y  (01 

) o 

YES 

002  C039604M  (M 

) 1 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCHLG116 

DESCRIPTION: 

8TH  GRADERS 

IN 

SUMMER 

PROGRAMS 

IN  NONE  OF  ABOVE? 

GRADES /ASSESSMENTS : 

N08 , SOS 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C039605 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

2 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

1 

001  C039605Y  (01 

) o 

YES 

002  C039605M  (M 

) 1 

MISSING 

CONDITIONING  VARIABLE  ID: 

SCHL0117 

DESCRIPTION: 

WHAT  % OF  8TH 

GRDS  HELD  BACK/ REPEAT  8TH  GRADE? 

GRADES /ASSESSMENTS : 

N08 , S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

C041901 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

6 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

5 

001  CQ41901A  (01 

} 00000 

0% 

002  C041901B  (02 

) 10000 

1-2% 

003  C041901C  (03 

) 01000 

3-5% 

004  C041901D  (04 

) 00100 

6-10% 

005  C041901E  (05 

) 00010 

MORE  THAN  10% 

006  C041901M  (M 

) 00001 

MISSING 

CONDITIONING  VARIABLE  ID 

SCHL0173 

DESCRIPTION: 

PERCENT  OF 

STUDENTS  WHO  PARTICIPATED  IN  THE  NATIONAL  SCHOOL  LUNCH  PROGRAM 

GRADES/ ASSESSMENTS : 

N04 , S04 , N08  , 

S08 , N12 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

NTLUNSC 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

2 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

1 

001  NATLUNCM  (M 

) 0 

MISSING 

002  NATLUNCH  (@ 

) 1 

PERCENT 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL; 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0174 

PERCENT  OF  STUDENTS  WHO  PARTICIPATED  IN  THE  NATIONAL  SCHOOL  LUNCH  PROGRAM 
N04 , S04 , N08 , S08,  N12 


NTLUNSC 

LINEAR 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  NATLUNCL  ( 0-100, M=0  ) 0.0  + 1.0*X 


LINEAR 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0175 

PERCENT  OF  STUDENTS  WHO  PARTICIPATED  IN  REMEDIAL  READING 
N04 , S04 , N08 , S08,  N12 

REMRDSC  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 
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001  REMREADM  (M 

002  REM  READ  (@ 


) 0 

> 1 


MISSING 

PERCENT 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0176 

PERCENT  OF  STUDENTS  WHO  PARTICIPATED  IN  REMEDIAL  READING 
N04,  S04 , N08 , S08,  N12 

REMRDSC  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

LINEAR  NUMBER  OF  INDEPENDENT  CONTRASTS: 


1 

1 


001  REMREADL  (0-100, M=0  ) 0.0  * 1.0*X 


LINEAR 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0177 

PERCENT  OF  STUDENTS  WHO  PARTICIPATED  IN  REMEDIAL  MATH 
NO 4 , S04 , N08,  S08,  N12 

REMMHSC  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  REKMATHM  (M  ) 0 MISSING 

002  REM  MATH  (0  ) 1 PERCENT 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0178 

PERCENT  OF  STUDENTS  WHO  PARTICIPATED  IN  REMEDIAL  MATH 
N04 , S04 , N08 , S08,  N12 

REMMHSC  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

LINEAR  NUMBER  OF  INDEPENDENT  CONTRASTS: 


1 

1 


001  REMMATHL  (0-100,M=0  ) 0.0  + 1.0*X 


LINEAR 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0179 

PERCENT  OF  STUDENTS  IN  4TH  WHO  PARTICIPATED  IN  THE  NATIONAL  SCHOOL  LUNCH  PROGRAM 
N04 , S04 

NTL'JNGR  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  NATLUN4M  (M  ) 0 

002  NATLUN4  (@  > 1 


MISSING 

PERCENT 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0180 

PERCENT  OF  STUDENTS  IN  4TH  WHO  PARTICIPATED  IN  THE  NATIONAL  SCHOOL  LUNCH  PROGRAM 
N04,  S04 

NTLUNGR  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  1 

LINEAR  NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  NATLUN4L  (0-100,M=0  ) 0.0  + 1.0*X 


LINEAR 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0181 

PERCENT  OF  STUDENTS  IN  4TH  WHO  PARTICIPATED  IN  REMEDIAL  READING 
N04 , S04 

REMRDGR  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

I 


001  REMREA4M  (M  ) 0 MISSING 

002  REMREAD4  >1  PERCENT 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


SCHL0182 

PERCENT  OF  STUDENTS  IN  4TH  WHO  PARTICIPATED  IN  REMEDIAL  READING 
N04,  S04 

REMRDGR  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

LINEAR  NUMBER  OF  INDEPENDENT  CONTRASTS: 


1 

1 


001  REMREA4L  (0-100,M^O  ) 0.0  + 1.0*X 


LINEAR 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  TD: 

TYPE  OF  CONTRAST: 


SCHL0183 

PERCENT  OF  STUDENTS  IN  4TH  WHO  PARTICIPATED  IN  REMEDIAL  MATH 
N04,  S04 

REMMHCR  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS. 


2 

1 


001  REMMAT4M  (M 

002  PEMMATH4  (@ 


) 0 
) 1 


MISSING 

PERCENT 


CONDITIONING  VARIABLE  ID:  SCHL0184 

DESCRIPTION:  PERCENT  OF  STUDENTS  IN  4TH 

GRADES /ASSESSMENTS:  N04 , S04 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  REMMHGR 

TYPE  OF  CONTRAST:  LINEAR 

001  REMMAT4L  (0-100, M=0  ) 0.0  + 1.0*X 


CONDITIONING  VARIABLE  ID:  SCHL0185 

DESCRIPTION:  PERCENT  OF  STUDENTS  IN  8TH 

GRADES /ASSESSMENTS:  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  NTLUNGR 

TYPE  OF  CONTRAST:  CLASS 

001  NATLUN8M  (M  ) 0 

002  NATLUN8  (@  ) 1 


CONDITIONING  VARIABLE  ID:  SCHL0186 

DESCRIPTION:  PERCENT  OF  STUDENTS  IN  RTH 

GRADES /ASSESSMENTS:  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  NTLUNGR 

TYPE  OF  CONTRAST:  LINEAR 

001  NATLUN8L  {0-100, M=0  ) 0.0  + 1.0*X 


CONDITIONING  VARIABLE  ID:  SCHL0187 

DESCRIPTION:  PERCENT  OF  STUDENTS  IN  8TH 

GRADES/ASSESSMENTS:  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  REMRDGR 

TYPE  OF  CONTRAST:  CLASS 

001  REMREA8M  (M  ) 0 

002  REMREAD8  (@  ) 3 


CONDITIONING  VARIABLE  ID:  SCHL0188 

DESCRIPTION:  PERCENT  OF  STUDENTS  IN  8TH 

GRADES/ASSESSMENTS:  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID : REMRDGR 

TYPE  OF  CONTRAST:  LINEAR 

001  REMREA8L  (0-100. M^O  ) 00  + 1.0*X 


CONDITIONING  VARIABLE  ID:  SCHL0189 

DESCRIPTION:  PERCENT  OF  STUDENTS  IN  8TH 

GRADES /ASSESSMENTS:  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  REMMHGR 

TYPE  OF  CONTRAST:  CLASS 

001  REMMAT8M  (M  ) 0 

002  REMMATH8  (0  > 1 


CONDITIONING  VARIABLE  ID:  SCHL0190 

DESCRIPTION:  PERCENT  OF  STUDENTS  IN  8TH 

GRADES /ASSESSMENTS:  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  REMMHGR 

TYPE  OF  CONTRAST:  LINEAR 

001  REMMAT8L  (0-100, M=0  ) 0.0  + 1.0*X 


WHO  PARTICIPATED  IN  REMEDIAL  MATH 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  1 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

LINEAR 

WHO  PARTICIPATED  IN  THE  NATIONAL  SCHOOL  LUNCH  PROGRAM 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

MISSING 

PERCENT 

WHO  PARTICIPATED  IN  THE  NATIONAL  SCHOOL  LUNCH  PROGRAM 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  1 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

LINEAR 

WHO  PARTICIPATED  IN  REMEDIAL  READING 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

MISSING 

PERCENT 

WHO  PARTICIPATED  IN  REMEDIAL  READING 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  1 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

LINEAR 

WHO  PARTICIPATED  IN  REMEDIAL  MATH 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

MISSING 

PERCENT 

WHO  PARTICIPATED  IN  REMEDIAL  MATH 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  1 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

LINEAR 


CONDITIONING  VARIABLE  ID:  TCHR0001 

DESCRIPTION:  WHAT  IS  YOUR  GENDER? 

GRADES /ASSESSMENTS;  N04 , S04,  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T055901 

TYPE  OF  CONTRAST:  CLASS 

001  T05S901A  (01  ) 00 

002  T05S901B  (02  ) 10 

003  T055901M  (M  ) 01 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

NUMBER  OF  INDEPENDENT  CONTRASTS:  2 

MALE 

FEMALE 

MISSING 


BEST  COPY  AVAILABLE 
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CONDITIONING 

VARIABLE  ID: 

TCHR0002 

DESCRIPTION: 

WHICH  BEST  DESRIBES  YOU? 

GRADES/ ASSESSMENTS : 

N04 , S04 , N08,  S08 

CONDITIONING 

VAR  LABEL: 

NAEP  ID: 

T056001 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  C F CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

001  T056001A 

1 01  ] 

l 000000 

WHITE 

002  TO 56 C GIB 

(C2  1 

1 100000 

BLACK 

003  T356001C 

IC3 

1 010000 

HISPANIC 

004  Z 05600 ID 

(04 

> 0010C0 

ASIAN/PACIFIC  AMERIC 

005  TJ56001 E 

(05 

) 000100 

AMER  IND/ALASKA  NATV 

T056001 F 

(06 

i 000010 

OTHER 

■O'*  TO 5600 IM 

(M 

) oooooi 

MISSING 

CONDITIONING 

VARIABLE  ID: 

TCHR0C03 

DESCRIPTION: 

YEARS  TAUGHT 

GRADES  ASSESSMENTS : 

N04,  S04 , N08 , S08 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

T040301 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

C01  7343301A  <01 

) 00000 

2 YEARS  OR  LESS 

:o2  t:<402cib  o: 

) 10000 

3-5  YEARS 

:G3  73403G1C  (03 

) 01000 

5-10  YEARS 

CO 4 T040301D  (04 

) 00100 

11-24  YEARS 

CCS  70403 3 IE  (05 

) 00010 

25  YEARS  OR  MORE 

-.‘06  TO 433C1M  M 

) 00001 

MISSING 

: CMI?I TINNING  VARIABLE  ID: 
RESCRIPT I ON: 

GRADES  ASSESSMENTS: 
CONDITIONING  VAP  LABEL: 


TCHR0004 

HOW  MANY  YRS  TOTAL  YOU  TAUGHT  MATH? 
N04,  S04 


NAEP  ID: 

TYPE  OF  CONTRAST: 

T056101 

CLASS 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

:o:  T056101A 

(01 

) ocooo 

2 YEARS  OR  LESS 

*02  70561 01 B 

(02 

) 10000 

3-5  YEARS 

003  TC56101C 

(03 

) 01000 

6-10  YEARS 

CC4  TC56101D 

(04 

) 00100 

11-24  YEARS 

T056101E 

(05 

> 00010 

25  YEARS  OR  MORE 

0 26  TD561C1M 

(M 

) 00001 

MISSING 

CONDITIONING 

VARIAELE  ID: 

TCHR0005 

DESCRIPTION: 

HOW  MANY 

YRS  TOTAL  YOU  TAUGHT  SCIENCE? 

1 FADES ■ ASSESSMENTS : 

N04 , S04 

NAEP  ID- 

\ Ar\  LAbbL : 

T056102 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

Ov'l  T >5  61  02 A 

101 

> 00000 

2 YEARS  OR  LESS 

:■*:  T056102B 

i 02 

) 100C0 

3-5  YEARS 

03  T056102C 

<03 

) 01000 

6-10  YEARS 

.:-'4  T056102D 

(04 

> 00100 

11-24  YEARS 

:C6  T 0 5 6 1 0 2 E 

' 05 

) 00010 

25  YEARS  OR  MORE 

t: 56i com 

< M 

) 00001 

MISSING 

6 

5 


6 

e; 


"UNPSTIONIN3  VARIABLE  ID:  TCHRG006 

'ESCRI PTICN : 

'RACES  ASSESSMENTS: 

■ • i.t  i n , n : n • ’/ah  label  : 


TYPE  TCHNG  CERT  IN  THIS  ST  IN  MAIN  FIELD? 
N04 , S04 , NO 8 , S08 


NAEP  ID: 

T056201 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

TYPE  ■ F ■'VNTFAST : 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

-1  7 ' S A 1 j 1 A 

( 0 1 

000000 

ADVANCED  PROFESSIONL 

j.'v  r’'-5fl01P 

(02 

1 100C00 

REGULAR /STANDARD  ST 

*„■*  TC56 1 r 1C 

tO? 

i 010000 

PROBATIONARY  STATE 

■ J 

it  4 

1 001000 

TEMPORARY/ PROVISIONL 

V 5 TC5*  c:  IE 

(05 

1 000100 

OTHER  THAN  STATE  CRT 

"|V  TC5  mC'J  1 F 

<C», 

1 000010 

NOT  HAVE  CERT  MAIN 

t;  5 6 2 mi 

( !•! 

1 000001 

MISSING 

NT  I T I ‘ NTN  ; 

VARIABLE  ID: 

TCHR0007 

PhNCFimCN: 

CERTIFICATION. 

ELEMENTARY 

OR  MIDDLE/ JUNIOR  HS  ED? 

: .FADES  AS.'F.C 

DM  ENTS  : 

NO 4 . S04.  NOS, 

SOR 

. IID-T  T I (.'fJINCi 
L'AEr  IT 

VAR  LABEL- 

1*04  0501 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 

TYPE  r '"NT 

R A''T 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

<■".  r 4o*,.  r,' 

i , 

> 000 

YES 

.*  i 4 ‘"  IN 

< ,* 

l 100 

NO 

. L 1 I ‘ 4 1 1 ' 

, ■ ' i 

) 01 P 

NOT  OFFERED  IN  STATE 

7 

6 


4 

3 


36: 
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004  T040501M  (M 


) 001 


MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0008 

DESCRIPTION:  DO  YOU  HAVE  CERTIFICATION 

GRADES /ASSESSMENTS:  N04,  S04,  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T040506 

TYPE  OF  CONTRAST:  CLASS 

001  T040506Y  {01  ) 000 

002  T040506N  {02  ) 100 

003  T040506C  (03  ) 010 

004  T040506M  (M  ) 001 


CONDITIONING  VARIABLE  ID:  TCHR0009 

DESCRIPTION:  DO  YOU  HAVE  CERTIFICATION 

GRADES /ASSESSMENTS:  N04,  S04,  NOB,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T040504 

TYPE  OF  CONTRAST:  CLASS 

001  T040504Y  (01  ) 000 

002  T040504N  (02  ) 100 

003  T040504C  (03  ) 010 

004  T040504M  (M  } 001 


IN  ELEMENTARY  MATH? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  4 

NUMBER  OF  INDEPENDENT  CONTRASTS:  3 

YES 

NO 

NOT  OFFERED  IN  STATE 
MISSING 


IN  JR  HIGH/SEC  MATH? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  4 

NUMBER  OF  INDEPENDENT  CONTRASTS:  3 

YES 

NO 

NOT  OFFERED  IN  STATE 
MISSING 


CONDITIONING  VARIABLE  10  TCHR0012 

DESCRIPTION:  CERTIFICATION,  OTS'ER 

GRADES /ASSESSMENTS:  N04 , S04 , N08,  SOB 

CONDITIONING  VAR  LABEL- 

NAEP  ID:  T040505 

TYPE  OF  CONTRAST:  CLASS 

001  T040505Y  (01  ) 000 

002  T040505N  (02  ) 100 

003  T040505C  (03  ) 010 

004  T040505M  (M  ) 001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  4 

NUMBER  OF  INDEPENDENT  CONTRASTS:  3 

YES 

NO 

NOT  OFFERED  IN  STATE 
MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0013 

DESCRIPTION:  HIGHEST  ACADEMIC  DEGREE 

GRADES /ASSESSMENTS:  N04,  S04 , N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T056301 

TYPE  OF  CONTRAST:  CLASS 

001  T056301A  (01  ) 0000000 

002  T056301B  (02  ) 1000000 

003  T056301C  (03  ) 0100000 

004  T056301D  (04  ) 0010000 

005  T0563 DIE  (05  ) 0001000 

006  T05630IF  (06  ) 0000100 

007  T056301G  (07  ) 0000010 

008  T056301M  (M  } 0000001 


YOU  HOLD? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  B 

NUMBER  OF  INDEPENDENT  CONTRASTS:  7 

HIGH  SCHOOL  DIPLOMA 
ASSOCIATES / VOCATIONL 
BACHELORS  DEGREE 
MASTER'S  DEGREE 
EDUCATION  SPECIALIST 
DOCTORATE 

PROFESSIONAL  DEGREE 
MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0014 

DESCRIPTION:  EDUCATION  UNDERGRAD  MAJOR 


GRADES / ASSESSMENTS : N04 , S04 , N08,  SOS 

CONDITIONING  VAR  LABEL: 

NAEP  ID-  T040701 

TYPE  OF  CONTRAST:  CLASS 

001  T040701Y  (01  ) 0 

002  T040701M  <M  ) 1 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

YES 

MISSING 


CONDITIONING  VARIABLE  ID : 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LA3EL : 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR001 5 

ELMENT  ED  UNDERGRAD  MAJOR 
N04 , S04 , NOB,  SOB 

T0407U6 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T040706Y  (01  ) 0 YES 

002  T040706M  (M  ) 1 MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHROO 1 6 

SEC  ED  UNDERGF/ ' MAJOR 
NO 4 , S04  , NOB , :.()« 

T040707 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


BEST  COPY  AVAILABLE 

3B9 


001  T040707Y  (01 

002  T040707M  (M 


) 0 
) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0017 

•WAS  YOUR  UNDERGRADUATE  MAJOR  MATH? 

N04,  S04 , N08 , SOB 

T040703  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T040703Y  (01  ) 0 

002  T040703M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHROO 18 

WAS  YOUR  UNDERGRADUATE  MAJOR  MATH  ED? 

N04 , S04,  N08 , S08 

T040704  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T040704Y  (01  ) 0 

002  T04Q704M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0023 

SPECIAL  EDUCATION  UNDERGRAD  MAJOR 
N04 , S04 , N08 , S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T040708 

CLASS 


001  T040708Y  (01  ) 0 

002  T040708M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0024 

BILINGUAL  ED/ESL  UNDERGRAD  MAJOR 
N04 , S04 , NOS,  S08 

T040709  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T040709Y  (01  ) 0 

002  T040709M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0025 

OTHER  UNDERGRAD  MAJOR 
N04,  S04 , N08,  S08 

T040705 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T04070SY  (01  ) 0 

002  T040705M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR002  6 

EDUCATION  GRAD  MAJOR 
N04 , S04 , N08,  S08 

T040801 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T040801Y  (01  ) 0 

002  T040801M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0027 

ELEMENTARY  ED  GRAD  MAJOR 
N04 , SO 4 , N08 , SOB 

T040807 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T040807Y  (01  ) 0 

002  T040807M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID. 
DESCRIPTION: 

GRADES,  ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 


TCHR002R 

SECONDARY  ED  GRAD  MAJOR 
N04,  S04 , N08 , SOS 

T0408C8 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 


TYPE  OF  CONTRAST : 


CLASS 


NUMBER  OF  INDEPENDENT  CONTRASTS: 


1 


001  T040808Y  (01 

002  T040808M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T040803Y  (01 

002  T040803M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T040804Y  (01 

002  T040804M  (M. 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T040809Y  (01 

002  T040809M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T040810Y  (01 

002  T040810M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T040811Y  (01 

002  T040811M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T040812Y  (01 

002  T040812M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T040813Y  (01 

002  T040813M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 


) 0 YES 

> 1 MISSING 


TCHR0029 

WAS  YOUR  GRADUATE  MAJOR  MATHEMATICS? 
N04 , S04 , N08,  S08 


T040803  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


) 0 YES 

) 1 MISSING 


2 

1 


TCHR0030 

WAS  YOUR  GRADUATE  MAJOR  MATH  ED? 
N04 , S04 , N08 , S08 


T040804 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


) 0 YES 

) 1 MISSING 


TCHR0035 

SPECIAL  ED  GRAD  MAJOR 
N04 , S04 , NOS , S08 

T040809 

CLASS 

) 0 
) 1 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 

YES 

MISSING 


TCHR0036 

BILINGUAL  GRAD  MAJOR 
N04 , S04,  N08 , S08 


T040810 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


) 0 YES 

) 1 MISSING 


TCHR0037 

ADMIN/ SUPERVISION  GRAD  MAJOR 
N04 , S04 , N08 , SOS 


T04081 1 TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS : 2 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


) 0 YES 

) 1 MISSING 


TCHR0038 

CURRICULUM/ INSTRUCTION  GRAD  MAJOR? 
N04 , S04 , N08.  S08 


T040812 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


) 0 YES 

) 1 MISSING 


2 

1 


TCHR0039 

COUNSELING  GRAD  MAJOR? 
N04 , S04 , N08 , SOB 


T040813 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


) 0 
) 1 


YES 

MISSING 


TCHR0040 

OTHER  GRAD  MAJOR 
N04 , 304  NOS,  S08 
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NAEP  ID-  T040805 

TYPE  OF  CONTRAST:  CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  T040805Y  (Cl  ) 0 

002  T040805M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0041 

NO  GRADUATE  STUDY 
N04 , S04,  NOB,  S08 

T040806 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  T04C806Y  (01  ) 0 

002  T040806M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0042 

UNDERGRAD /GRAD  MINOR  STUDY -EDUCATION 
N04 , S04,  N08,  SOB 

T056401  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056401Y  (01  ) 0 *ES 

002  T056401M  (M  ) 1 MISSING 


2 

i 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0043 

UNDERGRAD /GRAD  MINOR  STUDY- ELEMENTARY  ED 
NO 4 , S04 , NOB,  S08 

T056402  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS : 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056402Y  (01  ) 0 

002  T056402M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0044 

UNDERGRAD /GRAD  MINOR  STUDY -SECONDARY  ED 
N04,  S04 , NOS,  S08 

T056403  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056403Y  (01  ) 0 

002  T056403M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0045 

UNDERGRAD/GRAD  MINOR  STUDY-MATHEMATICS 
N04 , S04,  NOB,  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS : 


T056404 

CLASS 


001  T0S6404Y  (01  ) 0 

0C2  T056404M  (M  > 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0046 

UNDERGRAD/ GRAD  MINOR  STUDY -MATHEMATICS  ED 
N04  S04,  NOB,  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056405 

CLASS 


001  T056405Y  (01  ) 0 

002  T056405M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0051 

UNDERGRAD /GRAD  MINOR  STUDY-SPECIAL  ED 
N04 , S04 , N08 , SOB 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056406 

CLASS 


001  T056406Y  (01  > 0 

002  T056406M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID:  TCHR0052 

DESCRIPTION:  UNDERGRAD /GRAD  MINOR  STUDY-BILINGUAL  ED 

GRADES /ASSESSMENTS : N04  , S04  , N08,  S08 


3 


i £ 


334 


CONDITIONING  VAR  LABEL; 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T056407Y  (01 

002  T056407M  (M 


T056407 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS; 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0053 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


UNDERGRAD/GRAD  MINOR  STUDY-ADMIN  & SUPERVISION 
NO 4 , S04 , N08 , SOS 


T056408 

CLASS 


TOTAL  UMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056408Y  (01 

002  T056408M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0054 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


UNDERGRAD /GRAD  MINOR  STUDY -CURRICULUM  Sc  INSTRUC 
N04,  S04 , NOB,  SC8 


T056409 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056409Y  (01 

002  T056409M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID  TCHR0055 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL; 
NAEP  ID: 

TYPE  OF  CONTRAST: 


UNDERGRAD/GRAD  MINOR  STUDY -COUNSELING 
N04 , S04 , NOS,  S08 


T056410 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS; 


001  T056410Y  (01 

002  T056410M  (M 


YES 

MISSING 


:ONDITIONING  VARIABLE  ID:  TCHR0056 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


UNDERGRAD /GRAD  MINOR  STUDY -OTHER 
N04,  S04 . N08 , SOB 


T056411 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056411Y  (01 

002  T056411M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  7CHR00S7 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


UNDERGRAD /GRAD  MINOR  STUDY-NONE 
N04,  S04 , N08 , S08 


T056412 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056412Y  (01 

002  T056412M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0058 


DESCRIPT  3 ON: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


LAST  YR.  HOW  MUCH  TIME  IN  MATH/MATH  ED  SEM/WRKSHP? 
N04 , S04 , NOB,  S08 


T056501 

c:ass 


TOTAL  NUMBER  OF  SPECIFIED  CONTRAST! 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056501A  (01 

002  T056501B  (02 

003  T056501C  (03 

004  T056501D  (04 

005  T056501E  (05 

006  T056501M  (M 


) 0C000 
) 10000 
) 01000 
) 00100 
J 00010 
) 00001 


NONE 

LESS  THAN  6 HOURS 
6 -IT'  HOURS 
16-35  HOURS 
MORE  THAN  35  HOURS 
MISSING 


CONDITIONING  VARIABLE  ID-  TCHR0060 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


LAST  2 YRS , HOW  MANY  MATH /MATH  ED  UN IV  COURSES? 
N04,  S04,  MOB,  SOB 


TO 5 6 60 1 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056601A  (01 

002  T056601D  (02 


O 7 Q 

vj  3 ) 


BEST  COPY  AVAILABLE 
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003  T056601C  (03  ) 01000 

004  T056601D  (04  ) 00100 

005  T056601E  (05  ) 00010 

006  T056601M  (M  ) 00001 


TWO 

THREE 

FOUR  OR  MORE 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0062 

PAST  5 YRS , TAKEN  COURSES /IN  PRO  DEVP-TELECOMM  USE 
N04 , S04 , N08 , S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056701 

CLASS 


001  T056701Y  (01  ) 0 

002  T056701M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0063 

PAST  5 YRS,  TAKEN  COURSES/ IN  PRO  DEVP-TECH  USE 
N04,  S04,  N08,  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056702 

CLASS 


001  T056702Y  (01  > 0 

002  T056702M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0064 

PAST  5 YRS,  TAKEN  COURSES /IN  PRO  DEVP-COOP  INSTRCT 
N04,  S04,  N08,  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056703 

CLASS 


001  T056703Y  (01  ) 0 

002  T056703M  (M  ) 1 


YES 

KISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0065 

PAST  5 YRS.  COURSES/ IN  PRO  DEVLP- INTERDISP  INSTRCT 
N04,  S04 , N08 , SOS 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056704 

CLASS 


001  T056704Y  (01  > 0 

002  T056704M  (M  J 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0066 

PAST  5 YRS,  COURSES/ IN  PRO  DEVLP- PORTFOLIO  ASSMNT 
N04,  S04 , N08,  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056705 

CLASS 


001  T056705Y  (01  ) 0 

002  T056705M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0067 

PAST  5 YRS,  COURSES/ IN  PRO  DEVLP- PERF  BASED  ASSMNT 
N04,  S04,  NO 8 , S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056706 

CLASS 


001  T0567O6Y  (01  ) 0 

002  T056706M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  GF  CONTRAST: 


TCHR0068 

PAST  5 YRS.  COURSES /PRO  DEVLP -TEACH  HIGHORDER  THKG 
N04.  S04,  N08  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056707 

CLASS 


001  T0567P7Y  (01  ) 0 

00.\  TC56707M  (M  ! 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 


TCHR0C69 

PAST  5 YRS,  COURSES/ PRO  DEVLP -TEACH  DIFF  CUI T BKC-D 
N04,  SO 4 , NOB , S08 


C / Jf 
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NAEP  ID: 

TYPE  OF  CONTRAST: 


001  T056708Y  (01 

002  T056708M  (M 


T056708 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0070 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T056709Y  (01 

002  T056709M  (M 


PAST  5 YRS , COURSES/ PRO  DEVLP-TEACH  LEP  STUDENTS 
N04,  S04 , NOB , S08 


T056709 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0071 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T056710Y  (01 

002  T056710M  (M 


PAST  5 YRS,  COURSES/ PRO  DEVLP-TEACH  SPEC  NEED  STDS 
N04,  S04 , N08 , S08 


T056710 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0072 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T056711Y  (01 

002  T056711M  (M 


PAST  5 YRS,  COURSES/ PRO  DEVLP-CLASSRM  MNGMT/ORG 
N04 , S04 , N08 , S08 


T056711 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0073 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T056712Y  (01 

002  T056712M  (M 


PAST  5 YRS,  COURSES/PRO  DEVLP-OTHER  PROF  ISSUES 
N04 , S04 , N08,  S08 


T056712 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0074 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T056713Y  (01 

002  T056713M  (M 


PAST  5 YRS,  COURSES/ PRO  DEVLP-NONE  OF  ABOVE 
N04 , S04 , N08,  S08 


T056713 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0075 


DESCRIPTION: 

GRADES/ ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T041201A  (01 

002  T0412C1B  (02 

003  T041201C  (03 

004  T041201D  (04 

005  T041201M  (M 


AVAILABILITY  OF  RESOURCES 
N04 , S04 , NOS , S08 


T041201 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS. 

GET  ALL  RESOURCES 
GET  MOST  RESOURCES 
GET  SOME  RESOURCES 
DONT  GET  RESOURCES 
KISSING 


CONDITIONING  VARIABLE  ID:  TCHR0076 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ARE  CURRICULUM  SPECIALISTS  AVAILABLE  FOR  MATH? 
N04 , S04 


T041302 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T041302Y  (01 

002  T041302N  (02 

003  T041302M  (M 


YES 

NO 

MISSING 


BEST  COPY  AVAILABLE 


r,  ci 
«J  i 0 


CONDITIONING  VARIABLE  ID:  TCHR0077 

DESCRIPTION:  SCIENCE  CURRICULUM  SPECIALIST 


GRADES /ASSESSMENTS:  N04,  S04 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T041303 

TYPE  OF  CONTRAST:  CLASS 

001  T04I3O3Y  {01  ) 00 

002  T041303N  (02  ) 10 

003  T041303M  {M  ) 01 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  3 

NUMBER  OF  INDEPENDENT  CONTRASTS:  2 

YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID; 

TYPE  OF  CONTRAST: 


TCHR0078 

HOW  MANY  SCHOOL  HOURS  ARE  PREP  TIME  PER  WEEK? 

N04,  S04 , N08 , S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056801 

CLASS 


001 

T056801 A 

(01 

) 000000 

002 

T056801B 

(02 

) 100000 

003 

T056801C 

(03 

) 010000 

004 

T056801D 

(04 

) 001000 

005 

T05680IE 

(05 

) 000100 

006 

T056801F 

(06 

) 000010 

0C7 

T056801M 

(M 

) 000001 

NONE 

LESS  THAN  ONE 
1-2 
3-4 
5 

MORE  THAN  5 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION : 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0079 

METHODS  OF  TEACHING  ELEM  MATH-  1+ COLLEGE  COURSE 
N04,  S04 , N08 , S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T056901 

CLASS 


001  TO 56901 Y (01  ) 0 

002  T056901M  (M  ) 1 


YES 

MISSING 


7 

6 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0080 

METHODS  OF  TEACHING  ELEM  MATH -PART  COLLEGE  COURSE 
N04 , S04 , N08,  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T0569A1 

CLASS 


001  T0569A1Y  {01  ) 0 

002  T0569A1M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTFAST: 


TCHR0081 

METHODS  CF  TEACHING  ELEM  MATH -SEMINAR 
N04 , S04 , N08,  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T0569B1 

CLASS 


001  T0569BIY  (01  ) 0 

002  T0569B1M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES,  ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHRO  082 

METHODS  OF  TEACHING  ELEM  MATH- LITTLE  NO  EXPOSURE 
N04 , S04,  N08 , S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


T0569C1 

CLASS 


001  T0569C1Y  (01  > 0 

002  T0569C1M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0083 

NUMBER  SYSTEMS  L NUMERATION  - 1 ♦ COLLEGE  COURSE 
NO 4 , S04 , NOB.  S08 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS : 


T056902 

CLASS 


001  T056702Y  (01  ) 0 

1-02  T05*70:-f1  (M  > 1 


YES 

MISSING 


1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

CRAPES /ASSESSMENTS* 


TCHR00R4 

NUMBER  SYSTEMS  fc  NUMERATION- PART  COLLEGE  COURSE 
N04 , S04 , NOR , SOH 


3 


/ b 
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CONDITIONING  VAR  LABEL: 

NAEP  ID:  TO  569A2 

TYPE  OF  CONTRAST:  CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569A2Y  {01 

002  T0569A2M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0085 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


NUMBER  SYSTEMS  U NUMERATION- SEMINAR 
N04 , S04 , NOB , SOB 


T0569B2 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569B2Y  {01 

002  T0569B2M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0086 


DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


NUMBER  SYSTEMS  & NUMERATION -LITTLE /NO  EXPOSURE 
N04,  S04,  N08 , S08 


T0569C2 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569C2Y  {01 

002  T0569C2M  {M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0087 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


MEASUREMENT  IN  MATH-  1 +COLLEGE  COURSE 
N04 , S04,  N08 , S08 


T056903 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056903 Y {01 

002  T056903M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0088 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEJ 
NAEP  ID: 

TYPE  OF  CONTRAST: 


MEASUREMENT  IN  MATH- 
N04 , S04 , NOB,  S08 


T0569A3 

CLASS 


PART  COLLEGE  COURSE 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569A3Y  (01 

002  T0569A3M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0089 

MEASUREMENT  IN  MATH-  -SEMINAR 
N04 , S04 , NO 8 , S08 

T0569B3 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569B3Y  {01 

002  T0569B3M  (H 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0090 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


MEASUREMENT  IN  MATH- 
N04 , 504,  NO 8 , S08 


T0569C3 

CLASS 


•LITTLE  NO  EXPOSURE 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


C01  T0569C3Y  (01 
002  T0569C3K  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0091 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


GEOMETRY-  1 ♦COLLEGE  COURSE 
N04 , S04 , NOR,  SOR 


T056904 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0fj6904Y  (01 

002  T056904M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 


TCHR0092 

GEOMETRY- PART  COLLEGE  COURSE 
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nrnr  r\r\rw/  awaii  aoi  r 


GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


N04 , S04 , N08,  S08 

T0569A4 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  T0569A4 Y {01  ) 0 

002  T0569A4M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0093 

GEOMETRY -SEMINAR 
N04,  S04,  N08 , S08 

T0569B4 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  T0569B4Y  (01  ) 0 

002  T0569B4M  {M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0094 

GEOMETRY-LITTLE  NO  EXPOSURE 
N04 , S04 , N08 , S08 

T0569C4 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  2 

NUMBER  OF  INDEPENDENT  CONTRASTS:  1 


001  T0569C4Y  (01  ) 0 

002  T0569C4M  (M  > 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0095 

PROBABILITY/STATISTICS-  1 +COLLEGE  COURSE 
N04 , S04 , HO  8 , S08 

T056905  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


2 

1 


001  T056905Y  (01  ) 0 

002  T056905M  (M  ) 1 


YES 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0096 

PROBABILITY/ STATISTICS -PART  COLLEGE  COURSE 
N04 , S04,  N08 , S08 

T0569A5  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569A5Y  (01  ) 0 

002  T0S69A5M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0097 

PROBABILITY/ STATISTICS- SEMINAR 
N04 , S04 , NOB  f S08 

T0569B5 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569B5Y  (01  1 0 

002  T0569B5M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TCHR0098 

PROBABILITY /STATIST ICS -LITTLE  NO  EXPOSURE 
N04,  S04 , NOR,  S08 

T0569C5  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0S69C5Y  (01  ) 0 

C02  T0569C5M  (M  ) 1 


YES 

MISSING 


2 

1 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST : 


TCHR0099 

COLLEGE  ALGEBRA-  1+COLLEGE  COURSE 
N04 , S04 , NOR,  SOB 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS; 


T0S6906 

CLASS 


001  T05f.?06Y  (01  ) 0 

002  T0b69 06M  (M  ) l 


YES 

MISSING 


2 

I 


CONDITION  1 NO  VARIABLE  ID: 


TCHR0100 
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DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL : 
NAEP  ID: 

TYPE  OF  CONTRAST: 


COLLEGE  ALGEBRA- PART  COLLEGE  COURSE 
N04 , S04,  N08,  S08 


T0569A6 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569A6Y  (01 

002  T0569A6M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0101 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


COLLEGE  ALGEBRA- SEMINAR 
N04,  S04 , N08 , S08 


T0569B6 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569B6Y  (01 

002  T0569B6M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0102 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


COLLEGE  ALGEBRA- LITTLE  NO  EXPOSURE 
N04 , S04 , N08 , S08 


T0569C6 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569C6Y  (01 

002  T0569C6M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0103 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


CALCULUS-  1+ COLLEGE  COURSE 
N04 , S04 , N08 , S08 


T056907 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS, 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T056907Y  (01 

002  T056907M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0104 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


CALCULUS -PART  COLLEGE  COURSE 
N04 , S04 , N08 , S08 


T0569A7 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569A7Y  (01 

002  T0569A7M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0105 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


CALCULUS-SEMINAR 
N04,  S04 , NQ8 , S08 


T0569B7 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569B7Y  (01 

002  T0569B7M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0106 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


CALCULUS -LITTLE  NO  EXPOSURE 
N04 , S04 , NOB,  S08 


T0569C7 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS; 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569C7Y  (01 

002  T0569C7M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0107 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ABSTRACT/ LINEAR  ALGEBRA-  l+COLLEGE  COURSE 
N04 , S04,  NOB,  S08 


T056908 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T058908Y  {01 

002  T056908M  (M 


YES 

MISSING 


* ')  '*1  Q 
^ ) i O 
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CONDITIONING  VARIABLE  ID:  TCHR0108 


DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ABSTRACT/ LINEAR  ALGEBRA- PART  COLLEGE  COURSE 
N04,  S04 , NOS , S08 


T0569A8 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569A8Y  (01 

002  T0569A8M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0109 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ABSTRACT/ LINEAR  ALGEBRA-SEMINAR 
N04 , S04 , N08 , S08 


T0569B8 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569B8Y  (01 

002  T0569B8M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0110 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ABSTRACT/ LINEAR  ALGEBRA- LITTLE  NO  EXPOSURE 
N04,  S04 , N08 , S08 


T0569C8 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0569C8Y  1 01 

002  T0569C8M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHRG111 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


EVER  STUDIED  ESTIMATION? 
N04 , S04 , N08 , S08 


T057001 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057001Y  (01 

002  T057001N  (02 

003  T0S7001M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0112 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


EVER  STUDIED  PROBLEM  SOLVING  IN  MATH? 
N04,  S04 , NOB,  S08 


T057002 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057002Y  (01 

002  T057002N  (02 

003  T057002M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0113 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


EVER  STUDIED  USE  OF  MAN I PULAT IVES’ 
N04,  S04,  NOB,  SOB 


T057003 

CLASS 


TOTAL  NUMBER  Or  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  TO 5 7 00 3 Y (01 

002  TO 57 00 3N  (02 

003  T057003M  (K 


CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES ' ASSESSMENTS : 
CONDITIONING  VAF  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


EVER  STUDIED  USE  OF  CALCULATORS  IN  MATH  INSTRC’ 
N04 , S04 , NOB,  SCR 


TO  57004 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS : 


001  T057O04 Y (01 

002  T057004N  (02 

003  T0570C4M  <M 


YES 

NO 

MIS5I1N 


CONDITIONIN';  7AM Alili:  ll 
DESCRIPTION. 

GRADES /ASSESSMENTS 
CONDITIONING  VAR  LAPFL 
NAEP  ID: 

TYPE  ov  CONTRAST 


TUIP01 15 

EVER  STUL  .ED  UNDERS7 ANI.  IN  ; STMLS  MATH  THINK  IN: 
K04,  S04 , NOB,  SOR 


T057005 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


O 

KJ 
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001  T057005Y  {01 

002  T057005N  (02 

003  T057005M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0116 


DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


EVER  STUDIED  GENDER  ISSUES  IN  TEACHING  MATH? 
N04 , S04 , N08 , S08 


T057006 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057006Y  (01 

002  T057006N  (02 

003  T057006M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0117 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


EVER  STUDIED  TEACHING  STUDS  OF  DIFF  CULTURES? 
N04,  S04 , N08 , S08 


T057007 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057007Y  (01 

002  T057007N  (02 

003  T057007M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0118 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


KNOWLEDGE  OF  NCTM  CURR  k EVAL  STANDARDS  FOR  MATH? 
N04 , S04 , N08 , S08 


T057101 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T05710XA  (01 

002  T057101B  (02 

003  T057101C  (03 

004  T057101D  (04 

005  T057101M  (M 


VERY  KNOWLEDGEABLE 
KNOWLEDGEABLE 
SOMEWHAT  KNOWLEDGEAB 
LITTLE /NO  KNOWLEDGE 
MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0119 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


PRO  ACTVTS- STRATEGIES  LOCAL  WORKSHOPS 
NO  4 , S04 , N08 , S08 


T057201 

CLASS 


TOTAL  NUMBER  OF  SPECIFIEI  CONTRASTS: 
NUMBER  OF  INDEPENDENT  COl.TRASTS: 


001  T057201Y  (01 

002  T057201M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0120 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABE! 
NAEP  ID: 

TYPE  OF  CONTRAST: 


PRO  ACTVTS- STRATEGIES  REGIONAL  NCTM  MEETING 
N04 , S04 , N08,  S08 


T057211 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057211Y  (01 

002  T057211M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0121 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID; 

TYPE  OF  CONTRAST: 


PRO  ACTVTS- STRATEGIES  NATIONAL  NCTM  MEETING 
N04 . S04 , N08 , S08 


T057221 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057221Y  (01 

002  T057221M  (M 


YES 

MISSING 


CONDITIONING  VARIABLE  ID:  TCHR0122 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OK  CONTRAST: 


PRO  ACTVTS -STRATEGIES  OTHER 
N04 , S04,  N08,  SOB 


T057*  U 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRA  -TS: 


001  T057231Y  (01 

002  T057231M  (M 


YES 

MISSING 
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CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

TCHR0123 

PRO  ACTVTS- STRATEGIES  NO 

GRADES /ASSESSMENTS : 

NO 4 , S04 , NOB, 

S08 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

T057241 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

2 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

1 

OOl  T057241Y  (01  ) 

0 

YES 

002  T057241M  {M  ) 

1 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

TCHR0124 

YRS  TOTAL  TAUGHT  MATH 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

NOB,  S08 
T063001 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

6 

TYPE  OF  CONTPAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

5 

001  T063001A  (01  > 

00000 

2 YEARS  OR  LESS 

002  T063001B  (02  > 

lOOOO 

3-5  YEARS 

003  T063001C  (03  ) 

01000 

6-10  YEAi  3 

004  T06300 ID  (04  ) 

00100 

11-24  YEARS 

OOS  T063001E  (05  ) 

00010 

25  YEARS  OR  MORE 

006  T063001M  (M  ) 

00001 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

TCHR0125 

CURRICULUM  SPECIALIST  TO  HELP/ADVISE  IN  MATH? 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

N08 , SOB 
T058301 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

3 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

2 

001  T058301Y  (01  ] 

i 00 

YES 

002  T058301N  (02  1 

I 10 

NO 

003  T058301M  <M  1 

) 01 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

TSUB00C1 
IMPORTANCE  W/ 

STUDS -APPLYING  MATH  IDEAS? 

GRADES /ASSESSMENTS : 

N04,  S04 , N08 , 

S08 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

T057301 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

5 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

4 

001  T05730XA  (01 

) 0000 

VERY  IMPORTANT 

002  T057301B  (02 

) 1000 

SOMEWHAT  IMPORTANT 

003  T057301C  (03 

) 0100 

NOT  VERY  IMPORTANT 

004  T0573U1D  (04 

) 0010 

NOT  IMPORTANT 

005  T057301M  (M 

) 0001 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

TSUB0002 
IMPORTANCE  W/ 

STUDS -PROB  SOLVING=GOAL  U CONCEPT? 

GRADES /ASSESSMENTS : 

N04.  S04 , N08 , 

S08 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

T057302 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

5 

TYPE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

4 

001  TC573C2A  (01 

) 0000 

VERY  IMPORTANT 

002  T057302B  (02 

> 1000 

SOMEWHAT  IMPORTANT 

003  T057302C  (03 

) 0100 

NOT  VERY  IMPORTANT 

004  T057302D  (04 

) 0010 

NOT  IMPORTANT 

005  T057302M  (M 

1 0001 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

TSUB0003 
IMPORTANCE  W/ 

STUDS- 9 TECHS  PROMOTE  STUD  TALK? 

GRADES /ASSESSMENTS : 

N04,  S04,  NOB, 

, SOB 

CONDITIONING  VAR  LABEL: 
NAEP  ID: 

T057303 

TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

5 

TYFE  OF  CONTRAST: 

CLASS 

NUMBER  OF  INDEPENDENT  CONTRASTS: 

4 

001  TOS7303A  (01 

) 0000 

VERY  IMPORTANT 

002  T057303B  (02 

) 1000 

SOMEWHAT  IMPORTANT 

003  T057303C  (03 

) 0100 

NOT  VERY  IMPORTANT 

004  T057303D  (04 

> 0010 

NOT  IMPORTANT 

005  T057303M  (M 

) 0001 

MISSING 

CONDITIONING  VAR  I ABLE  ID  ■ TSUB0004 
DESCRIPTION: 

GRADES /ASSESSMENTS : 

CONDITIONING  VAR  LABEL: 

NAEP  ID: 

TYPE  OF  CONTRAST: 


IMPORTANCE  W STUDS-USE  RESULTS  TO  INFORM  DECISION 
N04 , S04,  NOS , SOS 


T057304 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 
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u v> 


001  T057304A  (01 

002  T057304B  (02 

003  T0S7304C  (03 

004  T057304D  (04 

005  T057304M  (K 


VERY  IMPORTANT 
SOMEWHAT  IMPORTANT 
NOT  VERY  IMPORTANT 
NOT  IMPORTANT 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0005 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TO  ACCESS  PROGRESS  HOW  OFTEN  USE  MULT  CHOICE  TESTS 
N04.  S04,  N08,  S08 


T057401 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057401A  (01 

002  T057401B  (02 

003  T057401C  (03 
C04  T057401D  (04 
005  T057401M  (M 


ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
ONCE  OR  TWICE  A YEAR 
NEVER /HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0006 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TO  ACCESS  PROGRESS  HOW  OFTEN  USE  PROBLEM  SETS 
N04 , S04,  N08 , S08 


T057402 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057402A  (01 

002  T057402B  (02 

003  T057402C  (03 

004  T057402D  (04 

005  T057402M  (M 


ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
ONCE  OR  TWICE  A YEAR 
NEVER/ HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0007 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TO  ACCESS  PROGRESS  HOW  OFTEN  USE  SHORT  WRITTEN  RSP 
N04 , S04 , N08,  S08 


T057403 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057403A  (01 

002  T057403B  (02 

003  T057403C  (03 

004  T057403D  (04 

005  T057403M  (M 


ONCE  OR  'TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
ONCE  OR  TWICE  A YEAR 
NEVER /HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0008 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TO  ACCESS  PROGRESS  HOW  OFTEN  USE  INDVL  GROUP  PRJCTS 
N04,  S04,  N08 , SOB 


T057404 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057404A  ( 

002  T057404B  (02 
C03  T057404C  (03 

004  T057404D  (04 

005  T057404M  (M 


ONCE  OR  TWICE  A WEEK 
ONCE  OR  TV/ICE  MONTH 
ONCE  OR  TWICE  A YEAR 
NEVER /HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0009 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TO  ACCESS  PROGRESS  HOW  OFTEN  USE  PORTFOl I OS 
N04 , S04 , NOS,  S08 


T0574C5 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057405A  (01 

002  T057405B  (02 

003  T057405C  (03 

004  T057405D  (04 

005  T057405M  (M 


) 0000 
> 1000 
) 0100 
) 0010 
) 0001 


ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
ONCE  OR  TWICE  A YEAR 
NEVER /HARDLY  EVER 
MISSiNG 


CONDITIONING  VARIABLE  ID:  TSUB0010 


DESCRIPTION : 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  t ABF.] 
NAEP  ID: 

TYPE  OF  CONTRAST: 


BEST  DESCRIPTION  OF  COMPUTE?  AVAILABLITY  IN  MATH 
N04 . S04,  NOB.  S0R 


T0575C1 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057501A  (01 

002  T057501B  (02 


) 000000 
) 100000 


NONE  AVAILABLE 
ONE  IN  CLASS 
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best  copy  AVAILABLE 

333 


003  T057501C  (03 

004  T057501D  (04 

005  T057501E  {05 

006  T057501F  (06 

007  T057501M  (M 


) 010000 
) 001000 
) 000100 
) 000010 
) 000001 


2 OR  3 IN  CLASS 
4 OR  MORE  IN  CLASS 
DIFFICULT  TO  ACCESS 
EASY  TO  ACCESS 
MISSING 


CONDITIONING  VARIA3LE  ID:  TSUB0011 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


PRIMARY  USE  OF  COMPUTERS  FOR  MATH  INSTRUCTION? 
N04,  S04 , N08 , S08 


T057601 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057601A  (01 

002  T057601B  (02 

003  T057601C  (03 

004  T057601D  (04 

005  T057601E  (05 

006  T057601M  (M 


) 00000 
) 10000 
) 01000 
) 00100 
} 00010 
) 00001 


DRILL  AND  PRACTICE 
DEMO  OF  NEW  TOPICS 
PLAYING  MATH  GAMES 
STIMULATIONS/ APPS 
DO  NOT  USE  COMPUTERS 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0012 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


ARE  STUDENTS  ASSIGNED  TO  THIS  CLASS  BY  ABILITY? 
N04 , S04 , NOR,  SOB 


T044002 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044002Y  (01 

002  T044002N  (02 

003  T044002M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0013 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


IF  ASSIGNED  BY  ABILITY, 
N04,  S04 , N08,  S08 


T057701 

CLASS 


WHAT  BASIS  ASSIGNED? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T0577C1A  (01 

002  TO 5 7 70 IB  (02 

003  T057701C  (03 

004  T057701D  (04 

005  T057701M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


NOT  BY  ABILITY 
BY  READING  ABILITY 
BY  MATH  ABILITY 
BY  GENERAL  ABILITY 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0014 


DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


IF  ASSIGNED  BY  ABILITY.  WHAT  IS  MATH  ABILITY? 
N04 , S04 , N08,  SOS 


T057801 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057801A  (01 

002  T057801B  (02 

003  T057B01C  (03 

004  T057R01D  (04 

005  T057801E  (05 

006  T057801M  (M 


} 00000 
) 10000 
) 01000 
) 00100 
) 00010 
) 00001 


NOT  BY  ABILITY 
HIGH  ABILITY 
AVERAGE  ABILITY 
LOW  ABILITY 
MIXED  ABILITY 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0015 


DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


CREATE  GROUPS  IN  CLASS  FOR  MATH  ON  ABILITY  BASIS? 
N04 , S04 , NOB.  S08 


T044201 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044201Y  (01 

002  T044201N  (02 

003  T044201M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0016 


DESCRIPTION . 

GRADES/ ASSESSMENTS ; 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TIME/WEEK  ON  MATH  INSTRUTION' 
N04,  S04 , N08 , SOB 


TO 4 4 3 01 
CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044301A  (01 

002  T044301B  (02 
00?  T044301C  (03 
004  T044301M  (M 


l 1 / 2 HOURS  OR  LESS 
'21/2  HRS-<  4 HRS 
4 HRS  OR  MORE 
MISSING 
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CONDITIONING  VARIABLE  ID:  TSUB0017 


DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  MUCH  TIME  PER  WEEK  STUDS  DO  MATH  W/  PARTNER? 
N04 , S04 , N08 , S08 


T0S790I 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T057901A  (01 
0C2  T057901B  (02 

003  T057901C  (03 

004  T057901D  (04 

005  T057901M  (M 


NONE 

LESS  THAN  1/2  HOUR 
1/2-1  HOUR 
MORE  THAN  1 HOUR 
MISSING 


CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


AMOUNT  MATH  HOMEWORK  ASSIGN /DAY? 
N04 , S04 , N08,  S08 


T044401 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  TO44401A  (01 

002  T044401B  (02 

003  T044401C  (03 

004  T044401D  (04 

005  T044401E  (05 

006  T044401F  (06 

007  T044401M  (M 


) 000000 
) 100000 
) 010000 
) 001000 
) 000100 
) 000010 
) 000001 


NONE 

15  MINUTES 
30  MINUTES 
4.5  MINUTES 
ONE  HOUR 

MORE  THAN  ONE  HOUR 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  OFTEN  DO  STUDENTS  DO  MATH  FROM  TEXTBOOKS? 
N04.  S04,  N08 , S08 


T044501 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044501 A (01 

002  T044501B  (02 

003  T044501C  (03 

004  T044501D  (04 

005  T044501M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB0020 

HOW  OFTEN  DO  STUDENTS  DO  MATH  ON  WORKSHEETS? 
N04 , S04 , N08 , SOB 


T044502 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044502A  (01 

002  T044502B  (02 

003  T044502C  (03 

004  T044502D  (04 

005  T044502M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB002 1 

HOW  OFTEN  DO  STUDENTS  SOLVE  PROBS  W/  OTHER  STUDS? 
N04 , SO 4 , N08,  S08 


T044512 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044512A  (01 

002  T0445I2B  (02 

003  T044512C  (03 

004  T044512D  (04 

005  T044512M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T044513A  (01 

002  T044M3B  (02 

003  T044513C  (03 

004  T044513D  (04 

005  T044S13M  (M 


TSUB0022 

HOW  OFTEN  DO  STUDENTS  WORK  W/  OBJECTS  LIKE  RULERS7 
N04,  S04 , N08 , S08 


T044513 

CLASS 

) 0000 
) 1000 
) OlUO 
) 0010 
) 0001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


3 JO 


rcqt  P.OPV  AVAILABLE 


CONDITIONING  VARIABLE  ID:  TSUB0023 

DESCRIPTION:  HOW  OFTEN  WORK  W/  COUNTING  BLOCKS . GEOMETRIC  SHAPES 

GRADES /ASSESSMENTS:  N04 , S04 , N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T044514 

TYPE  OF  CONTRAST:  CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044514A  (01  ) 0000  ALMOST  EVERY  DAY 

002  T044514B  (02  ) 1000  ONCE  OR  TWICE  A WEEK 

C03  T044514C  (03  ) 0100  ONCE  OR  TWICE  MONTH 

004  T044514D  (04  ) 0010  NEVER  OR  HARDLY  EVER 

005  T044514M  (M  ) 0001  MISSING 


5 

4 


CONDITIONING  VARIABLE  ID:  TSUB0024 

DESCRIPTION:  HOW  OFTEN  DO  STUDENTS 

GRADES /ASSESSMENTS:  N04.  S04 , N08,  SOB 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T044505 

TYPE  OF  CONTRAST:  CLASS 

001  T044505A  (01  ) 0000 

002  T044505B  (02  ) 1000 

003  T044505C  (03  > 0100 

004  T044505D  (04  ) 0010 

005  T044505M  (M  ) 0001 


ISE  A CALCULATOR? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVEP  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0C25 

DESCRIPTION:  HOW  OFTEN  DO  STUDENTS 

GRADES /ASSESSMENTS:  N04,  S04,  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T044515 

TYPE  OF  CONTRAST:  CLASS 

001  T044515A  (01  ) 0000 

002  T044515B  (02  ) 1000 

003  T044515C  (03  ) 0100 

004  T044515D  (04  ) 0010 

005  T044515M  (M  ) 0001 


TAXE  MATH  TESTS’ 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0026 

DESCRIPTION:  HOW  OFTEN  DO  STUDENTS 

GRADES/ ASSESSMENTS:  N04,  S04,  NOB , SOB 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T044507 

TYPE  OF  CONTRAST:  CLASS 

001  T044507A  (01  ) 0000 

002  T044507B  (02  ) 1000 

003  T044507C  (03  ) 0100 

004  T044507D  (04  ) 0010 

005  T044507M  (M  ) 0001 


WRITE  ABOUT  PROBLEM-SOLVING? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0027 

DESCRIPTION;  HOW  OFTEN  DO  STUDENTS 

GRADES /ASSESSMENTS:  N04,  S04,  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T044516 

TYPE  OF  CONTRAST:  CLASS 

001  T044516A  (01  ) 0000 

002  T044516B  (02  } 1000 

003  T044516C  (03  ) 0100 

004  T044516D  (04  ) 0010 

005  T044516M  (M  ) 0001 


TALK  ABOUT  MATH  WORK? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0028 

DESCRIPTION:  HOW  OFTEN  DO  STUDENTS 

GRADES/ ASSESSMENTS:  N04 , S04,  N08,  S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T044508 

TYPE  OF  CONTRAST:  CLASS 

001  T04450RA  (0L  ) 0000 

002  T04450RB  (02  > 1000 

003  T04450RC  (03  ) 0100 

004  T04450RD  (04  ) 0010 

005  T04450HM  ( M ) 0001 


WRITE  REPORTS/ DO  PROJECTS? 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


5 

4 


5 

4 


5 

4 


5 

4 


5 

4 


CONDITIONING  VARIABLE  ID:  TSUB0029 

DESCRIPTION:  HOW  OFTEN  DO  STUDENTS  DISCUSS  MATH  W/ OTHER  STDNTS’ 

GRADES / ASSESSMENTS : N04 , S04 , NOR,  SOB 


CONDITIONING  VAR  LABEL: 

NAEP  ID:  T044509 

TYPE  OF  CONTRAST:  CLASS 

001  T044509A  (01  ) 0000 

002  7044509B  (02  ) 1000 

003  T044509C  (03  ) 0100 

004  T044509D  (04  ) 0010 

005  T044509M  (M  > 0001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T044510A  (01 

002  T044510B  (02 

003  T044510C  (03 

004  T044510D  (04 

005  T044510M  (M 


TSUB0030 

HOW  OFTEN  DO  STUDENTS  WORK  REAL-LIFE  MATH  PRBLMS? 
N04 , S04 , NO 8 , SOB 


T044510 

CLASS 

) 0000 
) 1000 
) 0100 
) 0010 
) 0001 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  TO 4 4 5 0 6 A (01 

002  T044506B  (02 

003  T044506C  (03 

004  T044506D  (04 

005  T044506M  (M 


TSUB003 1 

HOW  OFTEN  DO  STUDENTS  USE  A COMPUTER' 
N04 . SO 4 , N08 , S08 


T044506 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

ALMOST  EVERY  DAY 
ONCE  OR  TWICE  A WEEK 
ONCE  OR  TWICE  MONTH 
NEVER  OR  HARDLY  EVER 
MISSING 


CONDITIONING  VARIABLE  ID:  T3UB0032 

DESCRIPTION:  IN  MATH 

GRADES /ASSESSMENTS:  N04,  S04 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T058001 

TYPE  OF  CONTRAST:  CLASS 

001  T058001A  (01  ) 0000 

002  T058001B  (02  ) 1000 

003  T058C01C  (03  ) 0100 

004  T058001D  (04  ) 0010 

005  T058001M  (M  ) 0001 


IN  MATH  CLASS  HOW  OFTEN  ADDRESS -NUMBERS  & OPS? 
NO 4 , S04 , NOB , S08 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 

A LOT 
SOME 
A LITTLE 
NONE 
MISSING 


CONDITIONING  VARIABLE  ID 
DESCRIPTION: 

GRADES /ASSESSMENTS: 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T058002A  (01 

002  T058002B  (02 

003  TC58002C  (03 

004  T058002D  (04 

005  T058002M  (M 


TSUB0033 

IN  MATH  CLASS  HOW  OFTEN  ADDRESS-MEASUREMENT? 
N04,  S04 , NO 8 , SOS 


T058002 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS : 

A LOT 
SOME 

A LITTLE 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID.  TSUB0034 

DESCRIPTION:  IN  MATH  CLASS  HOW  OFTEN  ADDRESS -GEOMETRY? 

GRADES /ASSESSMENTS:  N04 , S04 , NO 8 , S08 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T058003  TOTAL  NL 

TYPE  OF  CONTRAST : CLASS  NUMBER  C 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T058Q03A  (01 

002  T058003B  (02 

003  T058003C  (03 

004  T058003D  (04 

005  T058003M  (M 


A LOT 
SOME 

A LITTLE 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID 
DESCRIPTION : 

GRADES /ASSESSMENTS ; 
CONDITIONING  VAR  LABEL. 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSU13003  5 

IN  MATH  CLASS  HOW  OFTEN  ADPKE: 
NO 4 , S04 , NOR.  S0R 

T058004 

CLASS 


DATA  ANALYSIS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


349 


001 

T058004A 

(01 

) 0000 

A LOT 

002 

T058004B 

(02 

) 1000 

SOME 

003 

T058004C 

(03 

) 0100 

A LITTLE 

004 

T058004D 

(04 

) 0010 

NONE 

005 

T058004M 

(M 

) 0001 

MISSING 

CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB003  6 

IN  MATH  CLASS  HOW  OFTEN  ADDRESS -ALGEBRA  k FUNCT? 

N04,  S04 , N08 , S08 

T058005  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001 

T058005A 

(01 

) 0000 

002 

T058005B 

(02 

) 1000 

003 

T058005C 

(03 

) 0100 

004 

T058005D 

(04 

) 0010 

005 

T058005M 

(M 

) 0001 

A LOT 
SOME 

A LITTLE 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


T3UB0037 

IN  MATH  HOW  OFTEN  ADDRESS- LRN  MATH  FACTS /CONCEPTS? 

N04 , S04 , N08 , S08 

T058006  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001 

T058006A 

(01 

) 0000 

002 

T058006B 

(02 

) 1000 

003 

T058006C 

(03 

) 0100 

004 

T058006D 

(04 

) 0010 

005 

T058006M 

(M 

) 0001 

A LOT 
SOME 

A LITTLE 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB0038 

IN  MATH  HOW  OFTEN  ADDRESS -LRN  SKILLS/ PROCEDURES? 

N04,  S04 , NOS , S08 

T058007  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001 

T05B007A 

(01 

) 0000 

002 

T058007B 

(02 

) 1000 

003 

T058007C 

(03 

) 0100 

004 

T058007D 

(04 

) 0010 

005 

T058007M 

(M 

) 0001 

A LOT 
SOME 

A LITTLE 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB0039 

IN  MATH  HOW  OFTEN  ADDRESS-DEVELOP  REASONING  ABLTY? 

N04 , S04 , N08 , S08 

T058008  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001 

T058008A 

(01 

) 0000 

002 

T058008B 

(02 

) 1000 

003 

T058008C 

(03 

) 0100 

004 

T058008D 

(04 

) 0010 

005 

T058008M 

(M 

) 0001 

A LOT 
SOME 

A LITTLE 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB0040 

IN  MATH  HOW  OFTEN  ADDRESS -LRN  TO  COMMUNICATE  MATH? 

N04 , S04 , N08 , S08 

T058009  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


5 

4 


001 

T058009A 

(01 

) 0000 

002 

T058009B 

(02 

) 1000 

003 

T058009C 

(03 

) 0100 

004 

T058009D 

(04 

) 0010 

005 

T058009M 

(M 

) 0001 

A LOT 
SOME 

A LITTLE 

NONE 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB0041 

DO  YOU  PERMIT  UNRESTRICTED  USE  OF  CALCULATORS? 

N04 , S04 , N08 , S08 

T045401  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T045401 Y (01  ) 00 

002  T045401N  (02  > 10 


YES 

NO 


3 

2 


350 


t . . ■ 

a i ‘i 


OG 3 T045401M  (M 


} 01 


MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0042 


DESCRIPTION: 

GRADES / ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


DO  YOU  PERMIT  USE  OF  CALCULATORS  ON  TESTS? 
N04 , S04 , NOS,  SOB 


T044801 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS: 
NUMBER  OF  INDEPENDENT  CONTRASTS. 


001  T044801Y  (01 

002  T044801N  (02 

003  T044801M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TSUBOOO 


DESCRIPTION: 

GRADES /ASSESSMENTS . 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


STUDENTS  HAVE  ACCESS  TO  SCFL- OWNED  CALCULATORS? 
N04 , S04 , N08 , SOS 


T045001 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T045001Y  (01 

002  T04500 IN  (02 

003  T045001M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


TSUB0044 

DO  YOU  PROVIDE  INSTRUCTION  IN  USE  OF  CALCULATORS? 
N04 , S04 , NOS,  S08 


T044901 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T044901Y  (01 

002  T044901N  (02 

003  T044901M  (M 


YES 

NO 

MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0045 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  PREPARED  TO  TEACH  MATH  CONCEPTS? 
N04,  S04 , N08 , SOS 


T045304 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T045304A  (01 

002  T045304B  (02 

003  T045304C  (03 

004  T045304D  (04 

005  T04S304M  (M 


) 0000 
) 1000 
) 0100 
) 0010 
» 0001 


VERY  WELL  PREPARED 
MODERATELY  PREPARED 
NOT  VERY  PREPARED 
NOT  AT  ALL  PREPARED 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0046 


DESCRIPTION: 

GRADES /ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 


HOW  PREPARED  TO  TEACH  MATH  PROCEDURES'’ 
NO 4 , S04 , N08 , S08 


T045305 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS 
NUMBER  OF  INDEPENDENT  CONTRASTS: 


001  T045305A  (01 

002  T045305B  (02 

003  T045305C  (03 

004  T045305D  (04 

005  T045305M  (M 


) 0000 
» 1000 
) 0100 
1 0010 
i 0001 


VERY  WELL  PREPARED 
MODERATELY  PREPARED 
NOT  VERY  PREPARED 
NOT  AT  ALL  PREPARED 
MISSING 


CONDITIONING  VARIABLE  ID:  TSUB0047 

DESCRIPTION:  HOW  PREPARED  TO  TEACH  USF  OF  COMPUTERS’’ 

GRADES 'ASSESSMENTS : N04,  S04 , NOP,  SOB 

CONDITIONING  VAR  LABEL: 

NAEP  ID:  T045302  TOTAL 

TYPE  OF  CONTRAST:  CLASS  NUMBER 


TOTAL  NUMBER  OF  SPECIFIED  CONTRAST! 
NUMBER  OF  INDEPENDENT  CONTRASTS- 


001  T045302A  (01 

002  T045302B  (02 

003  T045302C  (01 

004  T045302D  (01 

005  T045302M  (M 


VERY  WELL  PREPARED 
MODERATELY  PREPARED 
NOT  VERY  PREPARED 
NOT  AT  ALL  PREPARED 
MISSING 


CONDITIONING  VAH I ABLE  T 1 J 
DESCRIPTION: 
GRADES/ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST- 


TSUB004R 

HOW  PREPARED  TO  TEA-  H IKK  Oh  CALCUIATORf ’ 
N04 , SO 4 , N08 , SO 8 


T045303 

CLASS 


TOTAL  NUMBER  OF  SPECIFIED  CONTRAST! 
NUMBER  OF  INDEPENDENT  CONTRASTS : 


U w J 


001  TQ45303A  (01 

002  T045303B  (02 

003  T045303C  (03 

004  T045303D  (04 

005  T045303M  (M 


CONDITIONING  VARIABLE  ID: 
DESCRIPTION: 

GRADES/ ASSESSMENTS : 
CONDITIONING  VAR  LABEL: 
NAEP  ID: 

TYPE  OF  CONTRAST: 

001  T044000A  (01 

002  T044000B  (02 

003  T044000C  (03 

004  T044000D  (04 

005  T044000E  (05 

006  T044000M  (M 


0000 

1000 

0100 

0010 

0001 


VERY  WELL  PREPARED 
MODERATELY  PREPARED 
NOT  VERY  PREPARED 
NOT  AT  ALL  PREPARED 
MISSING 


TSUB0049 

NUMBER  OF  STUDENTS  IN  CLASS? 

N04,  S04 , N08 , SOS 

TO 4 4000  TOTAL  NUMBER  OF  SPECIFIED  CONTRASTS:  6 

CLASS  NUMBER  OF  INDEPENDENT  CONTRASTS:  5 


00000 
> 10000 
) 01000 
) 00100 
) 00010 
) 00001 


1-20  STUDENTS 
21-25  STUDENTS 
26-30  STUDENTS 
31-35  STUDENTS 
36  OR  MORE  STUDENTS 
MISSING 


352 


Appendix  D 

IRT  PARAMETERS  FOR  MATHEMATICS  ITEMS 


This  appendix  contains  10  tables  of  IRT  (item  response  theory)  parameters  for  the  items 
that  were  used  in  each  mathematics  scale  for  the  fourth-  and  eighth-grade  State  Assessments. 

For  each  of  the  binary  scored  items  used  in  scaling  (i.e.,  multiple-choice  items  and  short 
constructed-response  items),  the  tables  provide  estimates  of  the  IRT  parameters  (which 
correspond  to  ah  bj,  and  q in  Equation  8.1  in  Chapter  8)  and  their  associated  standard  errors  (s.e.) 
of  the  estimates.  For  each  of  the  polytomously  scored  items  (i.e.,  the  extended  constructed- 
response  items  and  the  testlets),  the  tables  also  show  the  estimates  of  the  djv  parameters  (see 
Equation  8.1)  and  their  associated  standard  errors. 

The  tables  also  show  the  block  in  which  each  item  appears  for  each  grade  (Block)  and  the 
position  of  each  item  within  its  block  (Item). 

Note  that  because  the  item  parameters  in  this  appendix  are  in  the  metrics  used  for  the 
original  calibration  of  the  scales,  the  grade  4 and  grade  8 parameters  are  shown  in  different 
metrics.  The  transformations  needed  to  represent  these  parameters  in  terms  of  the  metric  of  the 
final  reporting  scales  are  given  in  Chapter  9. 


Table  D-l 

IRT  Parameters  for  Mathematics  Items 
Number  Sense , Properties , and  Operations  Scale , Grade  4 
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Table  D-l  (continued) 

IRT  Parameters  for  Mathematics  Items 
Number  Sense , Properties ; and  Operations  Scale,  Grade  4 
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Table  D-2 

IRT  Parameters  for  Mathematics  Items 
Measurement  Scale , Grade  4 
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Table  D-3 

IRT  Parameters  for  Mathematics  Items 
Geometry  and  Spatial  Sense  Scale , Grade  4 
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Table  D-4 

IRT  Parameters  for  Mathematics  Items 
Data  Analysis,  Statistics,  and  Probability  Scale,  Grade  4 
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Table  D-5 

IRT  Parameters  for  Mathematics  Items 
Algebra  and  Functions  Scale , Grade  4 
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IRT  Parameters  for  Mathematics  Items 
Number  Sense , Properties , and  Operations  Scale,  Grade  8 
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IRT  Parameters  for  Mathematics  Items 
Number  Sense,  Properties , and  Operations  Scale,  Grade  8 
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Table  D-7 

IRT  Parameters  for  Mathematics  Items 
Measurement  Scale , Grade  8 
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IRT  Parameters  for  Mathematics  hems 
Geometry ' and  Spatial  Sense  Scale,  Grade  8 
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Table  D-9 

IRT  Parameters  for  Mathematics  Items 
Data  Analysis,  Statistics,  and  Probability  Scale , Grade  8 
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Table  D-10 

IRT  Parameters  for  Mathematics  Items 
Algebra  and  Functions  Scale,  Grade  8 
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Appendix  E 


STATE  ASSESSMENT  PROGRAM  REPORTING  SUBGROUPS; 
COMPOSITE  AND  DERIVED  COMMON  BACKGROUND  VARIABLES 
AND  COMPOSITE  AND  DERIVED  REPORTING  VARIABLES 


REPORTING  SUBGROUPS  FOR  THE  1996  STATE  ASSESSMENT 

Results  for  the  1996  State  Assessment  were  reported  for  student  subgroups  defined  by 
gender,  race/ethnicity,  type  of  location,  parents’  level  of  education,  participation  in  the  National 
School  Lunch  Program,  and  eligibility  of  Title  I funding.  The  following  explains  how  each  of 
these  subgroups  was  derived. 


DSEX  (Gender) 

The  variable  SEX  is  the  gender  of  the  student  being  assessed,  as  taken  from  school 
records.  For  a few  students,  data  for  this  variable  was  missing  and  was  imputed  by  ETS  after  the 
assessment.  The  resulting  variable  DSEX  contains  a value  for  every  student  and  is  used  for 
gender  comparisons  among  students. 


DRACE  (Race/ethnicity) 

The  variable  DRACE  is  an  imputed  definition  of  race/ethnicity,  derived  from  up  to  three 
sources  of  information.  This  variable  is  used  for  race/ethnicity  subgroup  comparisons.  Two  items 
from  the  student  demographics  questionnaire  were  used  in  the  determination  of  derived 
race/ethnicity: 

Demographic  Item  Number  2: 

2.  If  you  are  Hispanic,  what  is  your  Hispanic  background? 

cd  I am  not  Hispanic 
cd  Mexican,  Mexican  American,  or  Chicano 
cd  Puerto  Rican 
co  Cuban 

cd  Other  Spanish  or  Hispanic  background 

Students  who  responded  to  item  number  2 by  filling  in  the  second,  third,  fourth,  or  fifth 
oval  were  considered  Hispanic.  For  students  who  filled  in  the  first  oval,  did  not  respond  to  the 
item,  or  provided  information  that  was  illegible  or  could  not  be  classified,  responses  to  item 
number  1 were  examined  in  an  effort  to  determine  race/ethnicity.  Item  number  1 read  as  follows: 


Demographic  Item  Number  1 : 

1 . Which  best  describes  you? 

cd  White  (not  Hispanic) 

cd  Black  (not  Hispanic) 

cd  Hispanic  (“Hispanic”  means  someone  who  is  Mexican,  Mexican  American, 
Chicano,  Puerto  Rican,  Cuban,  or  from  some  other  Spanish  or  Hispanic 
background.) 

cd  Asian  or  Pacific  Islander  (“Asian  or  Pacific  Islander”  means  someone  who  is 
Chinese,  Japanese,  Korean,  Filipino,  Vietnamese,  or  from  some  other  Asian  or 
Pacific  Island  background.) 

cd  American  Indian  or  Alaskan  Native  (“American  Indian  or  Alaskan  Native” 
means  someone  who  is  from  one  of  the  American  Indian  tribes,  or  one  of  the 
original  people  of  Alaska.) 

cd  Other  (What?) 


Students’  race/ethnicity  was  then  assigned  to  correspond  with  their  selection.  For 
students  who  filled  in  the  sixth  oval  (“Other”),  provided  illegible  information  or  information  that 
could  not  be  classified,  or  did  not  respond  at  all,  race/ethnicity  as  provided  from  school  records 
was  used. 

Derived  race/ethnicity  could  not  be  determined  for  students  who  did  not  respond  to 
background  items  1 or  2 and  for  whom  race/ethnicity  was  not  provided  by  the  school. 

An  exception  in  this  definition  of  race/ethnicity  was  made  for  Hawaii.  Students  from 
Hawaii  who  specified  Asian  or  Pacific  Islander  for  Demographic  Item  Number  1 were 
categorized  in  the  Asian  or  Pacific  Islander  race/ethnicity  classification,  no  matter  what  response 
they  gave  for  Demographic  Item  Number  2. 


TOL8  (Type  of  Location) 

TOL5 

TOL3 


The  variable  TOL8  is  used  by  NAEP  to  provide  information  about  the  type  of  location  in 
which  schools  are  located.  The  variable  is  defined  using  population  size  information  from  the 
1990  Census  and  the  definitions  of  Metropolitan  Statistical  Areas  (MS As)  as  of  June  1995.  There 
are  eight  categories  for  TOL8: 
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1 


2 

3 

4 

5 


Large  Central  City  a central  city  of  an  MSA  with  a population  greater  than  or  equal 

to  400,000,  or  a population  density  greater  than  or  equal  to  6,000 
persons  per  square  mile 


Midsize  Central  City  a central  city  of  an  MSA  but  not  designated  as  a large  city 

Urban  Fringe  of  a place  within  an  MSA  of  a large  central  city  and  defined  as 

Large  City  urban  by  the  U.S.  Bureau  of  Census 


Urban  Fringe  of  a a place  within  an  MSA  of  a midsize  central  city  and  defined 
Midsize  City  as  urban  by  the  U.S.  Bureau  of  Census 


Large  Town  a place  not  within  an  MSA,  but  with  a population  greater  than  or 

equal  to  25,000  and  defined  as  urban  by  the  U.S.  Bureau  of 
Census 


6 Small  Town 


7 Rural  MSA 

8 Rural  NonMSA 


a place  not  within  an  MSA,  with  a population  less  than  25,000, 
but  greater  than  or  equal  to  2,500  and  defined  as  urban  by  the 
U.S.  Bureau  of  Census 

a place  within  an  MSA  with  a population  of  less  than  2,500  and 
defined  as  rural  by  the  U.S.  Bureau  of  the  Census 

a place  not  within  an  MSA  with  a population  of  less  than  2,500 
and  defined  as  rural  by  the  U.S.  Bureau  of  the  Census 


The  variable  TOL5  collapses  the  information  provided  in  the  variable  TOL8  to  five 

levels: 

1 Large  Central  City 

2 Midsize  Central  City 

3 Urban  Fringe  of  Large  City,  Urban  Fringe  of  Midsize  City,  and  Large  Town 

4 Small  Town 

5 Rural  MSA  and  Rural  NonMSA 


The  variable  TOL3  is  used  extensively  in  the  NAEP  reports.  TOL3  collapses  TOLH  to 
three  levels: 

1 Central  City  (Large  Central  City  and  Midsize  Central  City)  This  category 

includes  central  cities  of  all  MSAs.  Central  City  is  a 
geographic  term  and  is  not  synonymous  with  “inner  city.’’ 

2 Urban  Fringe/Large  Town  (Urban  Fringe  of  Large  City,  Urban  Fringe  of  Midsize  City, 

and  Large  Town)  An  Urban  Fringe  includes  all  densely 
settled  places  and  areas  within  MSAs  that  are  classified  as 
urban  by  the  Bureau  of  the  Census.  A Large  Town  is  defined 


as  a place  outside  MSAs  with  a population  greater  than  or 
equal  to  25,000. 

3 Rural/Small  Town  (Small  Town,  Rural  MSA,  and  Rural  NonMSA)  Rural 

includes  all  places  and  areas  with  a population  of  less  than 
2,500  that  are  classified  as  rural  by  the  Bureau  of  the 
Census.  A Small  Town  is  defined  as  a place  outside  MSAs 
with  a population  of  less  than  25,000  but  greater  than  or 
equal  to  2,500. 


PARED  (Parents’  education  level) 

The  variable  PARED  is  derived  from  responses  to  two  questions,  B003501  and 
B003601,  in  the  student  demographic  questionnaire.  Students  were  asked  to  indicate  the  extent  of 
their  mother’s  education  (B003501 — How  far  in  school  did  your  mother  go?)  by  choosing  one  of 
the  following: 

cd  She  did  not  finish  high  school. 
cd  She  graduated  from  high  school. 
cd  She  had  some  education  after  high  school. 
cd  She  graduated  from  college. 
cd  I don’t  know. 

Students  were  asked  to  provide  the  same  information  about  the  extent  of  their  father’s 
education  (B003601 — How  far  in  school  did  your  father  go?)  by  choosing  one  of  the  following: 

cd  He  did  not  finish  high  school. 

cd  He  graduated  from  high  school. 

cd  He  had  some  education  after  high  school. 

cd  He  graduated  from  college. 

cd  I don’t  know. 

The  information  was  combined  into  one  parental  education  reporting  category  (PARED) 
as  follows:  If  a student  indicated  the  extent  of  education  for  only  one  parent,  that  le>el  was 
included  in  the  data.  If  a student  indicated  the  extent  of  education  for  both  parents,  the  higher  of 
the  two  levels  was  included  in  the  data.  For  students  who  did  not  know  the  level  of  education  for 
both  parents  or  did  not  know  the  level  of  education  for  one  parent  and  did  not  respond  for  the 
other,  the  parental  education  level  was  classified  as  unknown.  If  the  student  did  not  respond  for 
both  parents,  the  student  was  recorded  as  having  provided  no  response. 
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REGION  (Region  of  the  country) 


Results  for  each  state  were  compared  to  the  appropriate  regional  results  from  the 
national  component  of  the  assessment.  Jurisdictions  were  grouped  into  four  geographical 
regions — Northeast,  Southeast,  Central,  and  West — as  shown  in  Table  E- 1 . All  50  states  and  the 
District  of  Columbia  are  listed,  with  the  participants  in  the  State  Assessment  shown  in  italic  type. 
Territories  were  not  assigned  to  a region.  The  part  of  Virginia  that  is  included  in  the  Washington, 
DC,  metropolitan  statistical  area  is  included  in  the  Northeast  region;  the  remainder  of  the  state  is 
included  in  the  Southeast  region. 


Table  E-l 

NAEP  Geographic  Regions 


NORTHEAST 

SOUTHEAST 

CENTRAL 

WEST 

Connecticut 

Alabama 

Illinois 

Alaska 

Delaware 

Arkansas 

Indiana 

Arizona 

District  of 

Florida 

Iowa 

California 

Columbia 

Georgia 

Kansas 

Colorado 

Maine 

Kentucky 

Michigan 

Hawaii 

Maryland 

Louisiana 

Minnesota 

Idaho 

Massachusetts 

Mississippi 

Missouri 

Montana 

New  Hampshire 

North  Carolina 

Nebraska 

Nevada 

New  Jersey 

South  Carolina 

North  Dakota 

New  Mexico 

New  York 

Tennessee 

Ohio 

Oklahoma 

Pennsylvania 

Virginia 

South  Dakota 

Oregon 

Rhode  Island 

West  Virginia 

Wisconsin 

Texas 

Vermont 

Utah 

Virginia 

Washington 

Wyoming 

IEP  (Individualized  Education  Plan)1 

The  variable  IEP  comes  from  the  student  booklet  cover.  A value  of  1 indicates  that  a 
student  has  an  individualized  education  plan  while  a value  of  2 indicates  no  individualized 
education  program. 


1 A student  identified  on  the  Administration  Schedule  as  a student  with  a disability  (SD)  or  an  equivalent  classification 
may  be  excluded  from  the  assessment  il:  1 ) the  student  is  mainstreamed  less  than  50%  of  the  time  in  academic  subjects 
and  is  judged  incapable  of  participating  meaningfully  in  the  assessment,  or  2)  the  Individualized  Education  Plan  (IEP) 
team  or  equivalent  group  has  determined  that  the  student  is  incapable  of  paiticipating  meaningfully  in  the  assessment. 
SD/LEP  students  meeting  the  above  criteria  should  be  assessed  if,  in  the  judgment  of  school  staff,  they  are  capable  of 
taking  the  assessment. 
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LEP  (Limited  English  Proficiency) 


The  variable  LEP  comes  from  the  student  booklet  cover.  A value  of  1 indicates  that  a 
student  is  considered  to  have  limited  English  proficiency  while  a value  of  2 indicates  that  the 
student  does  not  have  limited  English  proficiency. 


TITLE1 

The  variable  TITLE  1 comes  from  the  student  booklet  cover.  A value  of  1 indicates  that  a 
student  is  eligible  for  Title  I funding  and  a value  of  2 indicates  that  the  student  is  not  eligible  for 
Title  1 funding. 


SLUNCH 

SLUNCH1 

The  variable  SLUNCH  is  provided  by  Westat,  Inc.  and  is  used  to  determine  if  a student 
participates  in  the  National  School  Lunch  Program.  The  values  for  this  variable  are  as  follow: 

1 not  eligible 

2 eligible  for  reduced  price  lunch 

3 eligible  for  free  lunch 

4 no  information  available 

5 school  refused  to  provide  information 


The  variable  SLUNCH  1 collapses  the  information  provided  in  the  variable  SLUNCH  to 
three  levels: 


1 eligible  for  free  or  reduced  price  lunch 

2 not  eligible  for  free  or  reduced  price  lunch 

3 no  information  available 


SCHTYPE 

The  variable  SCHTYPE  is  provided  by  Westat,  Inc.  and  is  used  to  determine  the  type  of 
school  that  a student  attended.  The  values  for  this  variable  are  as  follow: 

1 Public 

2 Private 

3 Catholic 

4 Bureau  of  Indian  Affairs  (BIA) 

5 Department  of  Defense  Education  Activity  (DoDEA)  schools 


VARIABLES  DERIVED  FROM  THE  STUDENT,  TEACHER,  AND  SCHOOL 
QUESTIONNAIRES 


Several  variables  were  formed  from  the  systematic  combination  of  response  values  for 
one  or  more  items  from  either  the  student  demographic  questionnaire,  the  student  mathematics 
background  questionnaire,  the  teacher  questionnaire,  or  the  school  questionnaire. 


HOMEEN2  (Home  environment — Articles  [of  4]  in  the  home) 


The  variable  HOMEEN2  was  created  from  the  responses  to  student  demographic  items 
B000901  (Does  your  family  get  a newspaper  regularly?),  B000903  (Is  there  an  encyclopedia  in 
your  home?),  B000904  (Are  there  more  than  25  books  in  your  home?),  and  B000905  (Does  your 
family  get  any  magazines  regularly?).  The  values  for  this  variable  were  derived  as  follows: 


1 

2 

3 

8 


0-2  types  The  student  responded  to  at  least  two  items  and  answered  Yes  to  two  or 
fewer. 


3 types 

4 types 
Omitted 


The  student  answered  Yes  to  three  items. 
The  student  answered  Yes  to  four  items. 
The  student  answered  fewer  than  two  items. 


VARIABLES  DERIVED  FROM  MATHEMATICS  ITEMS 
MATTAKE 

The  variable  MATTAKE  is  available  for  the  grade  8 sample.  This  variable  was  created 
from  items  M8 12501  through  M8 12507  in  order  to  provide  maximum  utility  of  this  data.  For 
some  analyses  this  variable  was  used  to  interpret  these  items  as  a single  response  item.  When  a 
student  responded  to  more  than  one  prompt  MATTAKE  was  coded  as  a multiple  response, 
however  when  the  student  responded  to  only  one  prompt  the  recoding  was  as  follows: 

1 I am  not  taking  mathematics  this  year 

2 Eighth-grade  mathematics 

3 Prealgebra 

4 Algebra 

5 Integrated  or  sequential  mathematics 

6 Applied  mathematics  (technical  preparation) 

7 Other  mathematics  class 


MATEXP 

The  variable  MATEXP  is  available  for  the  grade  8 sample.  This  variable  was  created 
from  items  M8 12601  through  M8 12609  in  order  to  provide  maximum  utility  of  this  data.  For 


some  analyses  this  variable  was  used  to  interpret  these  items  as  a single  response  item.  When  a 
student  responded  to  more  than  one  prompt  MATEXP  was  coded  as  a multiple  response, 
however  when  the  student  responded  to  only  one  prompt  the  recoding  was  as  follows: 

1 I do  not  expect  to  take  mathematics  class  in  ninth  grade 

2 Basic,  general,  business,  or  consumer  mathematics 

3 Applied  mathematics  (technical  preparation) 

4 Prealgebra 

5 Algebra  I or  elementary  algebra 

6 Geometry 

7 Integrated  or  sequential  mathematics 

8 Other  mathematics  class 

9 I don’t  know 


BKSCOR  (Booklet-level  score) 

The  booklet-level  score  is  a student-level  score  based  on  the  sum  of  the  number  correct 
for  dichotomous  items  plus  the  sum  of  the  scores  on  the  polytomous  items,  where  the  score  for  a 
polytomous  item  starts  from  0 for  the  unacceptable  category.  Thus,  for  a 4-point  extended 
constructed-response  item,  scores  of  “no  response”,  “off-task”,  and  “unsatisfactory”  are  assigned 
an  item  score  of  0.  Scores  of  “partial”,  “essential”,  and  “extensive”  are  assigned  item  scores  of  1 , 
2,  and  3,  respectively.  The  score  is  computed  based  on  all  cognitive  items  in  an  individual’s 
assessment  booklet. 


LOGIT  (Logit  percent  correct  within  booklet) 


In  order  to  compute  the  LOGIT  score,  a percent  correct  within  booklet  was  first 
computed.  This  score  was  based  on  the  ratio  of  the  booklet  score  (BKSCOR)  over  the  maximum 
booklet  score.  The  percent  correct  score  was  set  to  .0001  if  no  items  were  answered  correctly;  if 
BKSCOR  equaled  the  maximum  booklet  score,  the  percent  correct  score  was  set  to  .9999.  A logit 
score,  LOGIT,  was  calculated  for  each  student  by  the  following  formula: 


LOGIT  = In 


PCTCOR 
1 - PCTCOR 


LOGIT  was  then  restricted  to  a value  a\  such  that  -3  < a*  < 3.  After  computing  LOGIT  for  each 
student,  the  mean  and  standard  deviation  was  calculated  for  each  booklet  using  the  reporting 
sample  as  the  first  step  in  standardizing  the  logit  scores.  The  standardized  logit  score,  ZLOGIT, 
was  then  calculated  for  each  student  by  the  following  formula: 


ZSCORE  = 


LOGIT  - mean  LOGIT 
standard  deviation 
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NORMIT  (Normit  Gaussian  score) 

SCHNORM  (School-level  mean  Gaussian  score) 


The  normit  score  is  a student-level  Gaussian  score  based  on  the  inverse  normal 
transformation  of  the  mid-percentile  rank  of  a student's  number-correct  booklet  score  within  that 
booklet.  The  normit  scores  were  used  to  decide  collapsing  of  variables,  finalize  conditioning 
coding,  and  check  the  results  of  scaling. 

The  number-correct  is  based  on  the  number  of  dichotomous  items  answered  correctly 
plus  the  score  obtained  on  extended  constructed-response  items.  The  distributions  of  normit 
scores  were  constructed  from  the  reporting  sample  using  the  overall  reporting  weight.  The 
minimum  and  maximum  normits  were  set  io  -3.5  and  +3.5,  respectively.  The  mid-percentile  rank 
is  based  on  the  formula; 


CF(i)  + CF(i-l) 

2N 

where  CF(i)  is  the  cumulative  frequency  at  i items  correct  and  N is  the  total  sample  size.  If  i = 0 
then 


CF(0)  + 


CF(1) 


2N 


A school-level  normit,  SCHNORM  was  also  created;  this  was  the  mean  normit  across  all 
main  assessment  mathematics  booklets  administered  in  a school.  These  school-level  mean  normit 
scores  were  used  in  conditioning  procedures  to  take  into  account  differences  in  school 
proficiency.  For  each  school,  the  weighted  mean  (the  within  school  modular  student  weight, 
corrected  for  student  nonresponse)  of  the  normits  for  the  students  in  that  school  was  calculated. 
Each  student  was  then  assigned  that  mean  as  his  or  her  school-level  mean  normit  score  value. 


VARIABLES  RELATED  TO  PROFICIENCY  SCALING 
Proficiency  Score  Variables 

Item  response  theory  (IRT)  was  used  to  estimate  average  mathematics  proficiency  for 
each  jurisdiction  and  for  various  subpopulations,  based  on  students'  performance  on  the  set  of 
mathematics  items  they  received.  IRT  provides  a common  scale  on  which  performance  can  be 
reported  for  the  nation,  jurisdiction,  and  subpopulations,  even  when  all  students  do  not  answer 
the  same  set  of  questions.  This  common  scale  makes  it  possible  to  report  on  relationships 
between  students’  characteristics  (based  on  their  responses  to  the  background  questions)  and 
their  overall  performance  in  the  assessment. 


BEST  COPY  AVAILABLE 


A scale  ranging  from  0 to  500  was  created  to  report  performance  for  each  of  the  five 
mathematics  content  strands:  number  sense,  properties,  and  operations ; measurement',  geometry 
and  spatial  sense',  data  analysis,  statistics,  and  probabilities',  and  algebra  and  functions.  Each 
content  strand  scale  was  based  on  the  distribution  of  student  performance  across  all  three  grades 
assessed  in  the  1996  national  assessment  (grades  4,  8,  and  12)  and  had  a mean  of  250  and  a 
standard  deviation  of  50.  A composite  scale  was  created  as  an  overall  measure  of  students’ 
mathematics  proficiency.  The  composite  scale  was  a weighted  average  of  the  five  content  strand 
scales,  where  the  weight  for  each  content  strand  was  proportional  to  the  relative  importance 
assigned  to  the  content  strand  as  specified  in  the  mathematics  objectives.  An  additional  scale  was 
created  for  the  items  designed  to  measure  estimation  abilities.  Although  the  items  comprising 
each  scale  were  identical  to  those  used  for  the  national  program,  the  item  parameters  for  the  State 
Assessment  scales  were  estimated  from  the  combined  data  from  all  jurisdictions  participating  in 
the  State  Assessment. 

Scale  proficiency  estimates  were  obtained  for  all  students  assessed  in  the  State 
Assessment.  The  NAEP  methods  use  random  draws  (“plausible  values”)  from  estimated 
proficiency  distributions  to  compute  population  statistics.  Plausible  values  are  not  optimal 
estimates  of  individual  proficiency;  instead,  they  serve  as  intermediate  values  to  be  used  in 
estimating  population  characteristics.  Chapter  8 provides  further  details  on  the  computation  and 
use  of  plausible  values. 

The  proficiency  score  (plausible  value)  variables  are  provided  on  the  student  data  files 
for  each  of  the  scales  and  are  named  as  shown  in  Table  E-2. 


Table  E-2 

Scaling  Variables  for  the  1996  State  Assessment  Samples 


Mathematics  Scale 

Data  Variables 

Numbers  Sense,  Properties,  and 
Operations 

MRPSll  to 
MRPS15 

Measurement 

MRPS21  to 
MRPS25 

Geometry 

MRPS31  to 
MRPS35 

Data  Analysis,  Statistics,  and 
Probability 

MRPS41  to 
MRPS45 

Algebra  and  Functions 

MRPS51  to 
MRPS55 

Composite 

MRPCM1  to 
MRPCM5 
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QUALITY  EDUCATION  DATA  VARIABLES  (QED) 


The  data  files  contain  several  variables  obtained  from  information  supplied  by  Quality 
Education  Data,  Inc.  (QED).  QED  maintains  and  updates  annually  lists  of  schools  showing  grade 
span,  total  enrollment,  instructional  dollars  per  pupil,  and  other  information  for  each  school. 
These  data  variables  are  retained  on  both  the  school  and  student  files  and  are  identified  in  the 
data  layouts  by  “(QED)”  in  the  SHORT  LABEL  field. 

Most  of  the  QED  variables  are  defined  sufficiently  in  the  data  codebooks.  Explanations 
of  others  are  provided  below. 

ORSHPT  is  the  Orshansky  Percentile,  an  indicator  of  relative  wealth  that  specifies  the 
percentage  of  school-age  children  in  a district  who  fall  below  the  poverty  line. 

IDP  represents,  at  the  school  district  level,  dollars  per  student  spent  for  textbooks  and 
supplemental  materials.  The  range  code  for  instructional  dollars  spent  per  pupil  excluding 
teacher  salaries  are: 

0 = Unclassified 

1 = Under  $10 

2 = $10-49 

3 = $50-99 

4 = $100-149 

5 = $150-299 

6 = $300-399 

7 = $400-499 

8 = $500-999 

9 = $1000  + 

ADULTED  indicates  whether  or  not  adult  education  courses  are  offered  at  the  school 

site. 


URBAN  defines  the  school’s  urbanization:  urban  (central  city);  suburban  (area 
surrounding  central  city,  but  still  located  within  the  counties  constituting  the  metropolitan 
statistical  area);  or  rural  (area  outside  any  metropolitan  statistical  area). 
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SETTING  THE  NAEP  ACHIEVEMENT  LEVELS  FOR 
THE  1996  MATHEMATICS  ASSESSMENT 

Mary  Lyn  Bourque 

National  Assessment  Governing  Board 


Introduction 

Since  1 984,  NAEP  has  reported  the  performance  of  students  in  the  nation  and  for  specific 
subpopulations  on  a 0-to-500  proficiency  scale.  The  history  and  development  of  the  scale  and  the 
anchoring  procedure  used  to  interpret  specific  points  on  that  scale  is  described  elsewhere  in  this  report. 

The  achievement  levels  reported  in  1996  were  first  developed  in  1992  thiough  a process 
described  in  more  detail  in  the  following  sections  of  this  chapter.  The  levels  were  based  on  the 
mathematics  assessment  framework  and  item  pools  developed  for  the  1990  and  1992  assessments. 
Although  the  National  Assessment  Governing  Board  (NAGB)  updated  this  framework  in  preparation  for 
the  1996  assessment,  the  differences  in  test  and  item  specifications  were  not  large  enough  to  warrant  a 
new  mathematics  scale.  Therefore,  a decision  was  made  by  the  Board  to  retain  the  same  levels  as 
reported  for  the  1990  and  1992  assessments,  thus  allowing  a third  point  of  comparison  for  the  short-term 
mathematics  trend. 


History  of  the  Achievement  Levels  Development  1990  - 1992 

The  1988  legislation1  created  an  independent  board,  the  National  Assessment  Governing  Board 
(NAGB),  responsible  for  setting  policy  for  the  NAEP  program.  The  Board  has  a statutory  mandate  to 
identify  “appropriate  achievement  goals  for  each  . . . grade  in  each  subject  area  to  be  tested  under  the 
National  Assessment.”  Consistent  with  this  directive,  and  striving  to  achieve  one  of  the  primary  mandates 
of  the  statute  “to  improve  the  form  and  use  of  NAEP  results,”  the  Board  set  performance  standards 
(called  achievement  levels  by  NAGB)  for  the  National  Assessment  in  1990  and  again  in  1992.  The  1994 
legislation  (Public  Law  103-382)  continued  the  policy  Board  with  slightly  increased  membership,  and  a 
continued  mandate  to  set  student  performance  standards  on  each  age  and  grade  tested. 

The  1990  trial,  initiated  in  December  1989  with  the  dissemination  of  a draft  policy  statement 
(NAGB,  1989)  and  culminating  22  months  later  in  the  publication  of  the  NAGB  report,  The  Levels  of 
Mathematics  Achievement  (Bourque  & Garrison,  1991),  consisted  of  two  phases:  the  main  study  and  a 
replication-validation  study.  Although  there  were  slight  differences  between  the  two  phases,  there  were 
many  common  elements.  Both  phases  used  a modified  (iterative/empirical)  Angoff  (1971)  procedure  for 
arriving  at  the  levels;  both  focused  on  estimating  performance  levels  based  on  a review  of  the  1990 
NAEP  mathematics  item  pool;  and  both  phases  employed  a set  of  policy  definitions  for  Basic,  Proficient, 
and  Advanced  (NAGB,  1990)  as  the  criteria  for  making  the  item  ratings.  However,  the  1990  process  was 


'Public  Law  100-297.  (1988).  National  assessment  of  educational  progress  improvement  act  (Article  No.  USC  1221).  Washington. 
DC. 


no 


evaluated  by  a number  of  different  groups  (see  Hambleton  & Bourque,  1991)  who  identified  technical 
flaws  in  the  1990  process.  These  evaluations  influenced  NAGB’s  decision  to  set  the  levels  again  in  1992 
and  to  not  use  the  1990  levels  as  benchmarks  for  progress  toward  the  national  goals  during  the  coming 
decade.  However,  it  is  interesting  to  note  that  the  1990  and  1992  processes  produced  remarkably  similar 
results. 


In  September  1991  NAGB  contracted  with  American  College  Testing  (ACT)  to  convene  the 
panels  of  judges  that  would  recommend  the  levels  on  the  1992  NAEP  assessments  in  reading,  writing, 
and  mathematics.  While  the  1992  level-setting  activities  were  not  unlike  those  undertaken  by  NAGB  in 
1990,  there  were  significant  improvements  made  in  the  process  for  1992.  There  was  a concerted  effort  to 
bring  greater  technical  expertise  to  the  process:  The  contractor  selected  by  NAGB  has  a national 
reputation  for  setting  standards  in  a large  number  of  certification  and  licensure  exams;  an  internal  and 
external  advisory  team  monitored  all  the  technical  decisions  made  by  the  contractor  throughout  the 
process;  and  state  assessment  directors  periodically  provided  their  expertise  and  technical  assistance  at 
key  stages  in  the  project. 

Setting  achievement  levels  is  a method  for  setting  standards  on  the  NAEP  assessment  that 
identifies  what  students  should  know  and  be  able  to  do  at  various  points  along  the  proficiency  scale.  The 
initial  policy  definitions  of  the  achievement  levels  were  presented  to  panelists  along  with  an  illustrative 
framework  for  more  indepth  development  and  operationalization  of  the  levels.  Panelists  were  asked  to 
determine  descriptions/definitions  of  the  three  levels  from  the  specific  framework  developed  for  the 
NAEP  assessment  with  respect  to  the  content  and  skills  to  be  assessed.  The  operationalized  definitions 
were  refined  throughout  the  level-setting  process,  as  well  as  validated  with  a supplementary  group  of 
judges  subsequent  to  the  level-setting  meetings.  Panelists  were  also  asked  to  develop  a list  of  illustrative 
tasks  associated  with  each  of  the  levels,  after  which  sample  items  from  the  NAEP  item  pool  were 
identified  to  exemplify  the  full  range  of  performance  of  the  intervals  between  levels.  The  emphasis  in 
operationalizing  the  definitions  and  in  identifying  and  selecting  exemplar  items  and  papers  was  to 
represent  the  full  range  of  performance  from  the  lower  level  to  the  next  higher  level.  The  details  of  the 
implementation  procedures  are  outlined  in  the  remainder  of  this  appendix. 


Preparing  for  the  1992  Mathematics  Level-Setting  Meeting 

It  is  important  for  the  planning  of  any  standard-setting  effort  to  know  how  various  process 
elements  interact  with  each  other.  For  example,  panelists  interact  with  pre-meeting  materials,  the  meeting 
materials  (i.e.,  the  assessment  questions,  rating  forms,  rater  feedback,  and  so  forth),  each  other,  and  the 
project  staff.  All  of  these  elements  combine  to  promote  or  degrade  what  has  been  called  intrajudge 
consistency  and  interjudge  consensus  (Friedman  & Ho,  1990). 

Previous  research  has  conceptualized  the  effects  of  two  major  kinds  of  interaction:  people 
interacting  with  text  (Smith  & Smith,  1988)  and  people  interacting  with  each  other  (Curry,  1987; 
Fitzpatrick,  1989).  To  assess  the  effects  of  textual  and  social  interaction  and  adjust  the  standard  setting 
procedures  accordingly,  a pilot  study  was  conducted  as  the  first  phase  of  the  1992  initiative. 

Reading  was  chosen  as  the  single  content  area  to  be  pilot-tested  since  it  combined  all  of  the 
various  features  found  in  the  other  NAEP  assessments,  including  multiple-choice,  short  constructed- 
response,  and  extended  constructed-response  items.  The  pilot  study  provided  the  opportunity  to 
implement  and  evaluate  all  aspects  of  the  operational  plan — background  materials,  meeting  materials, 
study  design,  meeting  logistics,  staff  function,  and  participant  function. 


Ton 


The  overall  pilot  effort  was  quite  successful.  The  level-setting  process  worked  well,  and  the  pilot 
allowed  the  contractor  to  make  improvements  in  the  design  before  implementation  activities  began.  For 
example,  schedule  changes  were  made  that  allowed  the  panelists  more  time  to  operationalize  the  policy 
definitions  before  beginning  the  item-rating  task.  Also,  the  feedback  mechanisms  used  to  inform  panelists 
about  interjudge  and  intrajudge  consistency  data  were  improved  for  clarity  and  utility  to  the  entire 
process. 


The  Mathematics  Level-Setting  Panel 

Sixty-nine  panelists  representing  32  jurisdictions  (31  states  and  the  District  of  Columbia)  from 
the  424  nominees  were  invited  to  participate  in  the  level-setting  process.  They  represented  mathematics 
teachers  at  grades  4,  8,  and  1 2,  nonteacher  educators,  and  members  of  the  noneducator  (general  public) 
community.  The  group  was  balanced  by  gender,  race/ethnicity,  NAEP  regions  of  the  country,  community 
type  (low  SES,  not  low  SES),  district  size,  and  school  type  (public/private).  One  panelist  was  unable  to 
attend  due  to  a family  emergency,  resulting  in  68  participants:  24  at  grade  4 and  22  at  grades  8 and  12. 


Process  for  Developing  the  Achievement  Levels 

The  four-and-one-half  day  session  began  with  a brief  overview  of  NAEP  and  NAGB,  a 
presentation  on  the  policy  definitions  of  the  achievement  levels,  a review  of  the  NAEP  mathematics 
assessment  framework,  and  a discussion  of  factors  that  influence  item  difficulty.  The  purpose  of  the 
presentation  was  to  focus  panelists’  attention  on  the  mathematics  framework  and  to  emphasize  the  tact 
that  panelists’  work  was  directly  related  to  the  NAEP  assessment,  not  to  the  whole  domain  of 
mathematics. 

All  panelists  completed  and  self-scored  an  appropriate  grade-level  form  of  the  NAEP  assessment. 
The  purpose  of  this  exercise  was  to  familiarize  panelists  with  the  test  content  and  scoring  protocols 
before  beginning  to  develop  the  preliminary  operationalized  descriptions  of  the  three  levels. 

Working  in  small  groups  of  five  or  six,  panelists  expanded  and  operationalized  the  policy 
definitions  of  Basic,  Proficient,  and  Advanced  in  terms  of  specific  mathematical  skills,  knowledge,  and 
behaviors  that  were  judged  to  be  appropriate  expectations  for  students  in  each  grade,  and  were  in 
accordance  with  the  current  mathematics  assessment  framework. 

The  policy  definitions  are  as  follows: 

Basic  This  level,  below  proficient,  denotes  partial  mastery  of  the  knowledge 

and  skills  that  are  fundamental  for  proficient  work  at  each  grade — 4,  8, 
and  12. 

Proficient  This  central  level  represents  solid  academic  performance  for  each  grade 
tested — 4,  8,  and  12.  Students  reaching  this  level  have  demonstrated 
competency  over  challenging  subject  matter  and  are  well  prepared  for 
the  next  level  of  schooling. 


Advanced  This  higher  level  signifies  superior  performance  beyond  proficient 
grade- level  mastery  at  grades  4,  8,  and  12. 

The  small  groups  were  allowed  to  brainstorm  about  what  student  performance  should  be,  using 
the  framework  and  their  experience  in  completing  the  NAEP  assessment  as  guides2.  A comprehensive 
listing  of  grade-level  descriptors  was  developed,  and  panelists  were  asked  to  identify  the  five  that  best 
described  what  students  should  be  able  to  do  at  each  of  the  levels.  Those  descriptors  appearing  with  the 
greatest  frequency  were  compiled  into  a discussion  list  for  the  grade-level  groups.  Additions,  deletions, 
and  modifications  were  made  as  a result  of  discussions,  and  the  groups  reached  general  agreement  that 
the  final  list  of  descriptors  represented  what  students  should  be  able  to  do  at  each  achievement  level. 

Panelists  next  received  two  hours  of  training  in  the  Angoff  method.  Training  was  customized  to 
reflect  the  unique  item  formats  of  the  particular  subject  area  assessment.  Once  a conceptual  consensus 
was  reached  about  the  characteristics  of  marginally  acceptable  examinees  at  each  of  the  three  levels, 
practice  items  from  the  released  pool  were  rated  by  the  panelists  according  to  the  process  defined  in  the 
contractor’s  plan.  For  multiple-choice  and  short  constructed-response  items,  panelists  were  asked  to  rate 
each  item  for  the  expected  probability  of  a correct  response  for  a group  of  marginally  acceptable 
examinees  at  the  Basic,  Proficient,  and  Advanced  levels.  For  extended  constructed-response  items, 
panelists  were  asked  to  review  20  to  25  student  response  papers  and  select  three  papers,  one  for  each 
achievement  level,  that  typified  marginally  acceptable  examinee  performance  for  that  level. 

Following  training  in  the  Angoff  method,  the  judges  began  the  rating  process,  inspecting  and 
rating  each  item  in  the  pool  for  the  expected  probabilities  of  answering  the  item  correctly  at  each  level. 
Panelists  completed  three  rounds  of  item  ratings.  For  Round  1,  panelists  first  answered  the  items  in  each 
section,  then  reviewed  their  answers  using  scoring  keys  and  protocols.  This  process  helped  ensure  that 
panelists  would  be  thoroughly  familiar  with  each  item,  including  the  foils  and  scoring  rubrics,  before 
rating  the  items.  Panelists  provided  item  ratings/paper  selections  for  all  three  achievement  levels,  one 
item  at  a time,  for  all  the  items  in  a section,  then  proceeded  to  the  next  set  of  items,  for  which  the  process 
was  repeated.  During  Round  1,  panelists  used  their  lists  of  descriptors  and  other  training  materials  for 
guidance  in  the  rating  process. 

Following  Round  1,  item  response  theory  (IRT)  was  used  to  convert  the  rating  results3  for  each 
rater  to  a latent  ability  scale  represented  by  the  Greek  letter  theta  (8).  This  0 scale  was  the  same  scale 
used  to  calibrate  the  NAEP  items  evaluated  by  each  panelist.  In  order  to  provide  meaningful  feedback 
about  item  ratings,  a special  relative  scale  was  constructed,  which  was  a linear  transformation  of  the 
theta  scale  having  a mean  of  75  and  standard  deviation  of  15.  Before  Round  2 of  the  rating  process, 
panelists  were  given  interjudge  consistency  information  using  this  relative  scale.  This  information 
allowed  panelists  to  see  on  the  scale  where  their  individual  mean  item  ratings  were,  relative  to  the  mean 
for  the  group  and  to  the  means  for  other  panelists.  Reasons  for  extreme  mean  ratings,  including  the 
possibility  that  some  panelists  misinterpreted  the  item  rating  task,  were  discussed  briefly. 

Before  Round  2,  panelists  were  also  given  item  difficulty  data.  This  information  was  presented  as 
the  percentage  of  students  who  answered  each  item  correctly  during  the  actual  NAEP  administration,  for 


^he  panelists  also  reviewed  about  half  the  item  pool  (the  half  they  would  not  be  rating  later)  so  the  descriptors  could  be  further 
modified  if  appropriate. 

'Because  the  IRT  item  parameters  were  not  available  for  the  polytomously  scored  (extended  constructed-response)  items,  these  items 
(five  at  grade  4,  six  each  at  grades  8 and  1 2)  were  not  included  in  the  following  discussion  of  results. 


items  scored  “correct”  or  “incorrect”  (i.e.,  multiple-choice  and  short  constructed-response  items),  and  as 
the  percentage  of  students  receiving  scores  of  1,  2,  3,  and  4 for  the  extended  constructed-response  items4. 
Panelists  were  told  that  this  item  difficulty  information  should  be  used  as  a reality  check.  For  items  on 
which  item  ratings  differed  substantially  from  the  item  difficulty  value,  panelists  were  asked  to 
reexamine  the  item  to  determine  if  they  had  misinterpreted  the  item  or  misjudged  its  difficulty.  Results  of 
the  data  analysis,  and  panelists’  own  evaluations,  indicated  that  the  item  difficulty  information  was 
perceived  as  very  useful  but  had  little  impact  on  panelists’  ratings. 

For  Round  2,  panelists  reviewed  the  same  set  of  items  they  had  rated  in  Round  1 and,  using  the 
interjudge  consistency  information,  the  item  difficulty  information,  and  the  information  provided  prior  to 
Round  1,  they  either  confirmed  their  initial  item  ratings  or  adjusted  their  ratings  to  reflect  the  additional 
information.  About  one-third  of  Round  1 item  ratings  were  adjusted  during  Round  2. 

Following  Round  2,  panelists’  ratings  were  reanalyzed  and  additional  information  was  presented 
to  panelists  concerning  intrajudge  variability  prior  to  Round  3.  For  each  panelist,  the  intrajudge 
variability  information  consisted  of  those  items  that  they  had  rated  differently  than  items  having  similar 
difficulty,  taking  into  consideration  the  panelist’s  aggregated  item  ratings.  That  is,  the  panelists’ 
aggregated  item  ratings  were  converted  to  the  theta  (0)  scale.  All  items  rated  by  the  panelists  were  then 
analyzed  in  terms  of  the  panelist’s  achievement  level  (0)  in  comparison  to  actual  student  performance  on 
the  items.  The  observed  item  rating  from  each  panelist  was  contrasted  to  an  expected  item  rating.  Those 
items  with  large  differences  between  observed  and  expected  ratings  were  identified.  Panelists  were  given 
this  information  and  asked  to  review  each  of  these  items  and  decide  if  their  Round  2 ratings  still 
accurately  reflected  their  best  judgments  of  the  items.  The  intrajudge  consistency  data  was  to  be  used  to 
flag  items  for  reconsideration  in  the  final  round  of  rating. 

For  Round  3,  panelists  reviewed  the  same  set  of  items  they  rated  in  Rounds  1 and  2 using  both 
the  new  intrajudge  variability  information  and  the  information  made  available  during  Rounds  1 and  2.  In 
addition,  panelists  could  discuss,  within  their  small  groups,  ratings  of  specific  items  about  which  they 
were  unsure.  About  20  percent  of  the  item  ratings  were  adjusted  during  Round  3. 


Process  of  Selecting  Exemplar  Items 

Following  the  standard-setting  meeting,  a series  of  procedures  was  implemented  to  select 
exemplar  items.  First,  expected  and  empirical  p- values  were  computed  for  each  item  in  the  released  item 
pool.  Expected  p-values  were  based  on  predicted  performance  at  the  cut-off  score  for  each  achievement 
level  and  empirical  p-values  were  based  on  the  average  performance  of  all  students  responding  to  the 
item.  Items  that  did  not  have  expected  p-values  > 0.5 1 for  any  of  the  levels  were  deleted  from  the  item 
pool.  Second,  items  were  compared  to  the  operationalized  descriptions  of  the  levels.  Items'  that  did  not 
match  the  content  of  the  descriptions  were  deleted  from  the  item  pool.  Third,  the  remaining  items  were 
classified  as  possible  Basic.  Proficient,  or  Advanced  exemplars  based  on  content  match.  Fourth,  the 
validation  panel  reviewed  the  items  and  recommended  a set  of  items  to  serve  as  exemplars  for  the  levels. 
The  final  set  of  items  was  reviewed  and  approved  by  NAGB  at  their  May  1992  meeting.  These 
procedures  are  described  in  detail  below. 


The  percentages  presented  to  the  raters  summed  to  l(X)  percent,  but  this  excluded  the  percentages — around  80  percent,  in  some 
cases — of  students  who  wrote  responses  that  were  judged  to  be  "off-task."  those  who  "skipped"  that  question  and  continued  beyond 
that  question,  and  those  who.  apparently,  "never  reached"  that  question. 


Using  the  standard-setting  ratings,  expected  p- values  were  computed  for  each  item  at  the  cut 
point  for  each  achievement  level.  The  criteria  described  below  were  applied  to  the  scale-level  results  and 
an  analysis  was  conducted  to  delineate  items  that  could  serve  as  exemplars  for  each  achievement  level 
(Basic,  Proficient,  Advanced).  More  specifically,  for  an  item  to  be  chosen  as  a possible  exemplar  for  the 
Basic  achievement  level: 

1 . The  expected  p-value  for  students  at  the  cut  point  for  the  Basic  level  of  achievement 
had  to  be  greater  than  0.5 1 : 

2.  The  content  of  the  item  had  to  match  the  content  of  the  operationalized  description 
of  Basic;  and 

3.  The  empirical  p-value  for  the  item  had  to  be  higher  than  empirical  p-values  for  items 
selected  as  exemplars  for  the  Proficient  level. 

As  an  example: 


Grade  4 Basic  Level  Item  M022801 

Level 

Basic 

Proficient 

Advanced 

Scale  point 

211 

248 

280 

Expected  p-value 

0.70 

0.82 

0.94 

Empirical  p-value  = 0.52 

For  an  item  to  be  chosen  as  a possible  exemplar  for  the  Proficient  achievement  level: 

1.  The  expected  p-value  for  students  at  the  cut-off  score  for  the  Proficient  level  of 
achievement  had  to  be  greater  than  0.5 1 ; 

2.  The  content  of  the  item  had  to  match  the  content  of  the  operationalized  description 
of  Proficient;  and 

3.  The  empirical  p-value  for  the  item  had  to  be  lower  than  empirical  p-values  for  Basic 
exemplar  items,  but  higher  than  student  p-values  for  Advanced  exemplar  items. 

As  an  example: 


Grade  4 Proficient  Level  Item  M022001 

Level 

Basic 

Proficient 

Advanced 

Scale  point 

211 

248 

280 

Expected  p-value 

0.37 

0.58 

0.76 

Empirical  p-value  = 0.35 

For  an  item  to  be  chosen  as  a possible  exemplar  for  the  Advanced  achievement  level: 

1.  The  expected  p-value  for  students  at  the  cut-point  for  the  Advanced  level  of 
achievement  had  to  be  greater  than  0.5 1 ; 

2.  The  content  of  the  item  had  to  match  the  content  of  the  operationalized  description 
of  Advanced;  and 

3.  The  empirical  p-value  for  the  item  had  to  be  lower  than  empirical  p- values  for 
Proficient  exemplar  items. 

As  an  example: 


Grade  4 Advanced  Level  Item  M023101 


Level 

Basic 

Proficient 

Advanced 

Scale  point 

211 

248 

280 

Expected  p-value 

0.29 

0.43 

0.61 

Empirical  p-value  = 0.22 

The  analysis  procedures  described  above  yielded  3 1 items  as  possible  grade  4 exemplars,  43 
items,  as  possible  grade  8 exemplars,  and  37  items  as  possible  grade  12  exemplars,  as  follows: 


Possible  Exemplar  Items  by  Grade  and  Achievement  Level 


Grade 

Basic 

Proficient 

Advanced 

4 

9 

14 

8 

8 

23 

15 

5 

12 

14 

16 

7 

For  grade  4,  the  possible  exemplars  represented  49  percent  of  the  released  item  pool.  For  grades 
8 and  12,  the  possible  exemplars  represented  54  percent  of  the  released  item  pool  for  each  grade. 


Process  for  Validating  the  Levels 

Eighteen  mathematics  educators  participated  in  the  item  selection  and  content  validation  process. 
Ten  of  the  panelists  were  mathematics  teachers  who  had  participated  in  the  original  achievement  levels- 
setting  process  and  who  had  been  identified  as  outstanding  panelists  by  grade  group  facilitators  during 
this  meeting.  The  other  eight  panelists  represented  the  National  Council  of  Teachers  of  Mathematics,  the 
Mathematical  Sciences  Education  Board,  and  state-level  mathematics  curriculum  supervisors.  To  the 
extent  possible,  the  group  was  balanced  by  race/ethnicity,  gender,  community  type,  and  region  of  the 
country. 


The  two-and-one-half  day  meeting  began  by  briefing  panelists  on  the  purpose  of  the  meeting. 
They  first  reviewed  the  operationalized  descriptions  of  the  achievement  levels  for  consistency  with  the 
NAGB  policy  definitions  of  Basic,  Proficient,  and  Advanced  and  with  the  NAEP  Mathematics 
Objectives.  Next,  they  reviewed  the  operationalized  descriptions  of  the  achievement  levels  for  qualities 
such  as  within-  and  across-grade  consistency,  grade-level  appropriateness,  and  utility  for  increasing  the 
public’s  understanding  of  the  NAEP  mathematics  results.  Finally,  working  first  in  grade  level  (4,  8,  and 
12)  groups  of  six  panei;  *:s  each,  then  as  a whole  group,  panelists  revised  the  operationalized  descriptions 
to  provide  more  within-  and  across-grade  consistency  and  to  align  the  language  of  the  description  more 
closely  with  the  language  of  the  NCTM  Standards.  Both  the  original  descriptions  and  the  revised 
descriptions  are  included  later  in  this  appendix. 

On  the  third  day,  panelists  again  split  into  grade-level  groups  of  six  panelists  each  and  reviewed 
the  possible  exemplar  items.  The  task  was  to  select  a set  of  items,  for  each  achievement  level  for  their 
grade,  that  would  best  communicate  to  the  public  the  levels  of  mathematics  ability  and  the  types  of  skills 
needed  to  perform  in  mathematics  at  that  level. 

After  selecting  sets  of  items  for  their  grades,  the  three  grade-level  groups  met  as  a whole  group  to 
review  item  selection.  During  this  process,  cross-grade  items  that  had  been  selected  as  exemplars  by  two 
grade  groups  (three  such  items  were  selected  by  grade  groups  4 and  8)  were  assigned  to  one  grade  by 
whole-group  consensus.  In  addition,  items  were  evaluated  by  the  whole  group  for  overall  quality.  Two 
items  were  rejected  by  the  group  during  this  process  due  to  possible  bias.  This  process  yielded  14  items 
as  recommended  exemplars  for  grade  4,  1 1 items  as  recommended  exemplars  for  grade  8,  and  14  items  as 
recommended  exemplars  for  grade  1 2. 


Mapping  Panelists’  Ratings  to  the  NAEP  Seales 

The  process  of  mapping  panelists’  ratings  to  the  NAEP  scales  made  significant  use  of  item 
response  theory  (IRT).  IRT  provides  statistically  sophisticated  methods  for  determining  the  expected 
performance  of  examinees  on  particular  test  items  in  terms  of  an  appropriate  measurement  scale.  The 
same  measurement  scale  simultaneously  describes  the  characteristics  of  the  test  items  and  the 
performance  of  the  examinees.  Once  the  item  characteristics  are  set,  it  is  possible  to  precisely  determine 
how  examinees  are  likely  to  perform  on  the  test  items  at  different  points  of  the  measurement  scale. 

The  panelists’  ratings  of  the  NAEP  test  items  were  likewise  linked,  by  definition,  to  the  expected 
performance  of  examinees  at  the  theoretical  achievement  level  cut  points.  It  was  therefore  feasible  to  use 
the  IRT  item  characteristics  to  calculate  the  values  on  the  measurement  scale  corresponding  to  each 
achievement  level.  This  was  done  by  averaging  the  item  ratings  over  panelists  for  each  achievement  level 
and  then  simply  using  the  item  characteristics  to  find  the  corresponding  achievement  level  cut  points  on 
the  IRT  measurement  scale.  This  process  was  repeated  for  each  of  u . * !AEP  content  strands  within  each 
grade  (4,  8,  and  12). 

In  the  final  stage  in  the  mapping  process,  the  achievement  level  cut  points  on  the  IRT 
measurement  scale  were  combined  over  content  strands  and  rescaled  to  the  NAEP  score  scale.  Weighted 
averages  of  the  achievement  level  cut  points  were  computed.  The  weighting  constants  accounted  for  the 
measurement  precision  of  the  test  items  evaluated  by  the  panelists,  the  proportion  of  items  belonging  to 
each  NAEP  content  strand,  and  the  linear  NAEP  scale  transformation.  These  weighted  averages  produced 
the  final  cut  points  for  the  Basic,  Proficient,  and  Advanced  achievement  levels  within  each  grade. 


Evaluation  of  the  Mathematics  Levels 

The  1992  mathematics  achievement  levels  used  here  to  report  the  1996  NAEP  data  were 
evaluated  under  a Congressional  mandate  by  the  National  Academy  of  Education  (NAE).  A series  of 
research  studies  were  mounted  by  the  NAE  (National  Academy  of  Education,  1993)  to  look  at  various 
aspects  of  the  validity  of  the  level-setting  process,  and  the  levels  adopted  by  NAGB.  Severat  of  these 
studies  focused  specifically  on  the  mathematics  achievement  levels,  and  were  conducted  for  the 
Academy  by  staff  at  the  Learning  Research  and  Development  Center  at  the  University  of  Pittsburgh. 
Based  on  these  studies,  the  Academy’s  1993  policy  report  concluded  that  the  achievement  levels  in 
mathematics  were  flawed  and  should  not  be  continued.  The  more  recent  report  from  the  Academy 
(National  Academy  of  Education,  1997)  concluded  that  the  current  achievement  levels  raised  serious 
concerns  about  their  reliability  and  validity,  were  not  reasonable  (i.e.,  were  set  too  high),  and  in  the  final 
analysis,  should  be  abandoned  by  the  end  of  the  century. 

While  NAGB  did  not  agree  with  the  earlier  policy  report  and  continues  to  disagree  with  the  more 
recent  one,  and  while  the  Board’s  contractor  and  its  technical  advisors  do  not  believe  the  weight  of  the 
evidence  supported  the  NAEP  conclusions,  the  Board  believes  that  standards-based  reporting  is 
responsive  to  the  needs  of  the  users  of  NAEP  data,  is  an  important  aspect  of  the  national  reform 
movement,  and  assists  in  making  NAEP  data  more  useful  and  understandable  to  the  public.  The  Board  is 
committed  to  making  improvements  in  the  process,  and  will  continue  to  support  further  investigation  into 
the  validity  of  the  levels  through  additional  research. 
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Figure  F-l 

Final  Description  of  1992  Mathematics  Achievement  Levels 

GRADE 4 

The  NAEP  content  strands  include:  (1)  number  sense,  properties,  and  operations;  (2)  measurement; 
(3)  geometry;  (4)  data  analysis,  statistics,  and  probability;  (5)  algebra  and  functions.  (Note:  At  the  fourth- 
grade  level,  algebra  and  functions  are  treated  in  informal  and  exploratory  ways,  often  through  the  study  of 
patterns.)  Skills  are  cumulative  across  levels — from  Basic  to  Proficient  to  Advanced. 


BASIC.  Fourth-grade  students  performing  at  the  basic  level  should  show  some  evidence  of  understanding 
the  mathematical  concepts  and  procedures  in  the  five  NAEP  content  strands. 

Specifically,  fourth  graders  performing  at  the  basic  level  should  be  able  to  estimate  and  use  basic 
facts  to  perform  simple  computations  with  whole  numbers;  show  some  understanding  of  fractions  and 
decimals;  and  solve  simple  real-world  problems  in  all  NAEP  content  strands.  Students  at  this  level  should 
be  able  to  use  though  not  always  accurately — four-function  calculators,  rulers,  and  geometric  shapes. 
Their  written  responses  are  often  minimal  and  presented  without  supporting  information. 


PROFICIENT.  Fourth-grade  students  performing  at  the  proficient  level  should  consistently  apply 
integrated  procedural  knowledge  and  conceptual  understanding  to  problem  solving  in  the  five  NAEP 
content  strands. 

Specifically,  fourth  graders  performing  at  the  proficient  level  should  be  able  to  use  whole  numbers 
to  estimate,  compute,  and  determine  whether  results  are  reasonable.  They  should  have  a conceptual 
understanding  of  fractions  and  decimals;  be  able  to  solve  real-world  problems  in  all  NAEP  content  strands; 
and  use  four-function  calculators,  rulers,  and  geometric  shapes  appropriately.  Students  performing  at  the 
proficient  level  should  employ  problem-solving  strategies  such  as  identifying  and  using  appropriate 
information.  Their  written  solutions  should  be  organized  and  presented  both  with  supporting  information 
and  explanations  of  how  they  were  achieved. 


ADVANCED.  Fourth-grade  students  performing  at  the  advanced  level  should  apply  integrated  procedural 
knowledge  and  conceptual  understanding  to  complex  and  nonroutine  real-world  problem  solving  in  the  five 
NAEP  content  strands. 

Specifically,  fourth  graders  performing  at  the  advanced  level  should  be  able  to  solve  complex  and 
nonroutine  real-world  problems  in  all  NAEP  content  strands.  They  should  display  mastery  in  the  use  of 
four-function  calculators,  rulers,  and  geometric  shapes.  These  students  are  expected  to  draw  logical 
conclusions  and  justify  answers  and  solution  processes  by  explaining  why,  as  well  as  how,  they  were 
achieved.  They  should  go  beyond  the  obvious  in  their  interpretations  and  be  able  to  communicate  their 
thoughts  clearly  and  concisely. 


388 


Figure  F-l  (continued) 

Final  Description  of  1992  Mathematics  Achievement  Levels 


GRADE 8 

NAF.P  content  strands:  (1)  number  sense,  properties,  and  operations;  (2)  measurement;  (3) 
geometry;  (4)  data  analysis,  statistics,  and  probability;  (5)  algebra  and  functions.  Skills  are  cumulative 
across  all  levels — from  Basic  to  Proficient  to  Advanced. 

BASIC.  Eighth-grade  students  performing  at  the  basic  level  should  exhibit  evidence  of  conceptual  and 
procedural  understanding  in  the  five  NAEP  content  strands.  This  level  of  performance  signifies  an 
understanding  of  arithmetic  operations — including  estimation — on  whole  numbers,  decimals,  fractions,  and 
percents. 

Eighth  graders  performing  at  the  basic  level  should  complete  problems  correctly  with  the  help  of 
structural  prompts  such  as  diagrams,  charts,  and  graphs.  They  should  be  able  to  solve  problems  in  all  NAEP 
content  strands  through  the  appropriate  selection  and  use  of  strategies  and  technological  tools — including 
calculators,  computers,  and  geometric  shapes.  Students  at  this  level  also  should  be  able  to  use  fundamental 
algebraic  and  informal  geometric  concepts  in  problem  solving. 

As  they  approach  the  proficient  level,  students  at  the  basic  level  should  be  able  to  determine  which 
of  available  data  are  necessary  and  sufficient  for  correct  solutions  and  use  them  in  problem  solving. 
However,  these  eighth  graders  show  limited  skill  in  communicating  mathematically. 

PROFICIENT.  Eighth-grade  students  performing  at  the  proficient  level  should  apply  mathematical 
concepts  and  procedures  consistently  to  complex  problems  in  the  five  NAEP  content  strands. 

They  should  be  able  to  conjecture,  defend  their  ideas,  and  give  supporting  examples.  They  should 
understand  the  connections  between  fractions,  percents,  decimals,  and  other  mathematical  topics  such  as 
algebra  and  functions.  Students  at  this  level  are  expected  to  have  a thorough  understanding  of  basic-level 
arithmetic  operations — an  understanding  sufficient  for  problem  solving  in  practical  situations. 

Quantity  and  spatial  relationships  in  problem  solving  and  reasoning  should  be  familiar  to  them,  and 
they  should  be  able  to  convey  underlying  reasoning  skills  beyond  the  level  of  arithmetic.  They  should  be 
able  to  compare  and  contrast  mathematical  ideas  and  generate  their  own  examples.  These  students  should 
make  inferences  from  data  and  graphs;  apply  properties  of  informal  geometry;  and  accurately  use  the  tools 
of  technology.  Students  at  this  level  should  understand  the  process  of  gathering  and  organizing  data  and  be 
able  to  calculate,  evaluate,  and  communicate  results  within  the  domain  of  statistics  and  probability. 


Figure  F-l  (continued) 

Final  Description  of  1992  Mathematics  Achievement  Levels 


ADVANCED.  Eighth-grade  students  at  the  advanced  level  should  be  able  to  reach  beyond  the  recognition, 
identification,  and  application  of  mathematical  rules  in  order  to  generalize  and  synthesize  concepts  and 
principles  in  the  five  NAEP  content  strands. 

They  should  be  able  to  probe  examples  and  counter  examples  in  order  to  shape  generalizations  from 
which  they  can  develop  models.  Eighth  graders  performing  at  the  advanced  level  should  use  number  sense 
and  geometric  awareness  to  consider  the  reasonableness  of  an  answer.  They  are  expected  to  use  abstract 
thinking  to  create  unique  problem-solving  techniques  and  explain  the  reasoning  processes  underlying  their 
conclusions. 

GRADE  12 

NAEP  content  strands:  (1)  number  sense,  properties,  and  operations;  (2)  measurement;  (3) 
geometry;  (4)  data  analysis,  statistics,  and  probability;  (5)  algebra  and  functions.  Skills  are  cumulative 
across  levels — from  Basic  to  Proficient  to  Advanced. 


BASIC.  Twelfth-grade  students  at  the  basic  level  should  demonstrate  procedural  and  conceptual 
knowledge  in  solving  problems  in  the  five  NAEP  content  strands. 

They  should  be  able  to  use  estimation  to  verify  solutions  and  determine  the  reasonableness  of 
results  as  applied  to  real-world  problems.  They  are  expected  to  use  algebraic  and  geometric  reasoning 
strategies  to  solve  problems.  Twelfth  graders  performing  at  the  basic  level  should  recognize  relationships 
presented  in  verbal,  algebraic,  tabular,  and  graphical  forms;  and  demonstrate  knowledge  of  geometric 
relationships  and  corresponding  measurement  skills. 

Twelfth  graders  at  the  basic  level  should  be  able  to  apply  statistical  reasoning  in  the  organization 
and  display  of  data  and  in  reading  tables  and  graphs.  They  also  should  be  able  to  generalize  from  patterns 
and  examples  in  the  areas  of  algebra,  geometry,  and  statistics.  At  this  level,  they  should  use  correct 
mathematical  language  and  symbols  to  communicate  mathematical  relationships  and  reasoning  processes; 
and  use  calculators  appropriately  to  solve  problems. 


PROFICIENT.  Twelfth-grade  students  at  the  proficient  level  should  consistently  integrate  mathematical 
concepts  and  procedures  to  the  solutions  of  more  complex  problems  in  the  five  NAEP  content  strands. 

Twelfth  graders  performing  at  the  proficient  level  should  demonstrate  an  understanding  of 
algebraic,  statistical,  and  geometric  and  spatial  reasoning.  They  should  be  able  to  perform  algebraic 
operations  involving  polynomials;  justify  geometric  relationships;  and  judge  and  defend  the  reasonableness 
of  answers  as  applied  to  real-world  situations.  These  students  should  be  able  to  analyze  and  interpret  data  in 
tabular  and  graphical  form;  understand  and  use  elements  of  the  function  concept  in  symbolic,  graphical,  and 
tabular  form;  and  make  conjectures,  defend  ideas,  and  give  supporting  examples. 
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Figure  F-l  (continued) 

Final  Description  of  1992  Mathematics  Achievement  Levels 


ADVANCED.  Twelfth-grade  students  at  the  advanced  level  should  consistently  demonstrate  the 
integration  of  procedural  and  conceptual  knowledge  and  the  synthesis  of  ideas  in  the  five  NAEP  content 
strands . 


They  should  understand  the  function  concept;  and  be  able  to  compare  and  apply  the  numeric, 
algebraic,  and  graphical  properties  of  functions.  They  should  apply  their  knowledge  of  algebra,  geometry, 
and  statistics  to  solve  problems  in  more  advanced  areas  of  continuous  and  discrete  mathematics. 

Twelfth  graders  performing  at  the  advanced  level  should  be  able  to  formulate  generalizations  and 
create  models  through  probing  examples  and  counterexamples.  They  are  expected  to  communicate  their 
mathematical  reasoning  through  the  clear,  concise,  and  correct  use  of  mathematical  symbolism  and  logical 
thinking. 


Figure  F-2 

Draft  Descriptions  of  the  Achievement  Levels 
Prepared  by  the  Original  Level-Setting  Panel 

Fourth-Grade  Draft  Descriptions 

BASIC.  The  Basic  level  signifies  some  evidence  of  conceptual  and  procedural  understanding  in  the 
five  NAEP  content  strands  of  number  sense , properties,  and  operations ; measurement;  geometry  and 
spatial  sense;  data  analysis , statistics,  and  probability;  and  algebra  and  functions.  Understanding  simple 
facts  and  single-step  operations  are  included  at  this  level,  as  is  the  ability  to  perform  simple  computations 
with  whole  numbers.  This  level  shows  a partial  mastery  of  estimation,  basic  fractions,  and  decimals  relating 
to  money  or  the  number  line;  it  shows  an  ability  to  solve  simple  real-world  problems  involving 
measurement,  probability,  statistics,  and  geometry.  At  this  level,  there  is  a partial  mastery  of  tools  such  as 
four-function  calculators  and  manipulatives  (geometric  shapes  and  rulers).  Written  responses  are  often 
minimal,  perhaps  with  a partial  response  and  lack  of  supportive  information. 

PROFICIENT.  The  Proficient  level  signifies  consistent  demonstration  of  the  integration  of 
procedural  knowledge  and  conceptual  understanding  as  applied  to  problem  solving  in  the  five  NAEP 
content  strands  of  number  sense,  properties,  and  operations;  measurement ; geometry  and  spatial  sense; 
data  analysis , statistics , and  probability ; and  algebra  and  functions.  The  Proficient  level  indicates  an  ability 
to  perform  computation  and  estimation  with  whole  numbers,  to  identify  fractions,  and  to  work  with 
decimals  involving  money  or  the  number  line.  Solving  real-world  problems  involving  measurement, 
probability,  statistics,  and  geometry  is  an  important  part  of  this  level.  This  level  signifies  the  ability  to  use, 
as  tools,  four-function  calculators,  rulers,  and  manipulatives  (geometric  shapes).  It  includes  the  ability  to 
identify  and  use  pertinent/appropriate  information  in  problem  settings.  The  ability  to  make  connections 
between  and  among  skills  and  concepts  emerges  at  this  level.  Clear  and  organized  written  presentations, 
with  supportive  information,  are  typical.  And,  there  is  an  ability  to  explain  how  the  solution  was  achieved. 

ADVANCED,  The  Advanced  level  signifies  the  integration  of  procedural  knowledge  and 
conceptual  understanding  as  applied  to  problem  solving  in  the  five  NAEP  content  strands  of  number  sense, 
properties,  and  operations;  measurement;  geometry  and  spatial  sense;  data  analysis,  statistics,  and 
probability;  and  algebra  and  functions.  This  is  evidenced  by  divergent  and  elaborate  written  responses.  The 
Advanced  level  indicates  an  ability  to  solve  multistep  and  nonroutine  real-world  problems  involving 
measurement,  probability,  statistics,  and  geometry,  and  an  ability  to  perform  complex  tasks  involving 
multiple  steps  and  variables.  Tools  are  mastered,  including  four-function  calculators,  rulers,  and 
manipulatives  (geometric  shapes).  This  level  signifies  the  ability  to  apply  facts  and  procedures  by 
explaining  why  as  welt  as  how.  Interpretations  extend  beyond  obvious  connections  and  thoughts  are 
communicated  clearly  and  concisely.  At  this  level,  logical  conclusions  can  be  drawn  and  complete 
g justifications  can  be  provided  for  answers  and/or  solution  processes. 
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Figure  F-2  (continued) 

Draft  Descriptions  of  the  Achievement  Levels 
Prepared  by  the  Original  Level-setting  Panel 


Eighth-Grade  Draft  Descriptions 

BASIC.  Basic  students  should  begin  to  describe  objects,  to  process  accurately  and  elaborate 
relationships,  to  compare  and  contrast,  to  find  patterns,  to  reason  from  graphs,  and  to  understand  spatial 
reasoning.  This  level  of  partial  mastery  signifies  an  understanding  of  arithmetic  operations  on  whole 
numbers,  decimals,  fractions,  and  percents,  including  estimation.  Problems  that  are  already  set  up  are 
generally  solved  correctly,  as  are  one-step  problems.  However,  problems  involving  the  use  of  available  data, 
and  determinations  of  what  is  necessary  and  sufficient  to  solve  the  problem,  are  generally  quite  difficult. 
Students  should  select  appropriate  problem-solving  tools,  including  calculators,  computers,  and 
manipulatives  (geometric  shapes)  to  solve  problems  from  the  five  content  strands.  Students  should  also  be 
able  to  use  elementary  algebraic  concepts  and  elementary  geometric  concepts  to  solve  problems.  This  level 
indicates  familiarity  with  the  general  characteristics  of  measurement.  Students  at  this  level  may  demonstrate 
limited  ability  to  communicate  mathematical  ideas. 


PROFICIENT.  Proficient  students  apply  mathematical  concepts  consistently  to  more  complex 
problems.  They  should  make  conjectures,  defend  their  ideas,  and  give  supporting  examples.  They  have 
developed  the  ability  to  relate  the  connections  between  fractions,  percents,  and  decimals,  as  well  as  other 
mathematical  topics.  The  Proficient  level  denotes  a thorough  understanding  of  the  arithmetic  operations 
listed  at  the  Basic  level.  This  understanding  is  sufficient  to  permit  applications  to  problem  solving  in 
practical  situations.  Quantity  and  spatial  relationships  are  familiar  situations  for  problem  solving  and 
reasoning,  and  this  level  signifies  an  ability  to  convey  the  underlying  reasoning  skills  beyond  the  level  of 
arithmetic.  Ability  to  compare  and  contrast  mathematical  ideas  and  generating  examples  is  within  the 
Proficient  domain.  Proficient  students  can  make  inferences  from  data  and  graphs;  they  understand  the 
process  of  gathering  and  organizing  data,  calculating  and  evaluating  within  the  domain  of  statistics  and 
probability,  and  communicating  the  results.  The  Proficient  level  includes  the  ability  to  apply  the  properties 
of  elementary  geometry.  Students  at  this  level  should  accurately  use  the  appropriate  tools  of  technology. 


ADVANCED.  The  Advanced  level  is  characterized  by  the  ability  to  go  beyond  recognition, 
identification,  and  application  of  mathematical  rules  in  order  to  generalize  and  synthesize  concepts  and 
principles.  Generalization  often  takes  shape  through  probing  examples  and  counterexamples  and  can  be 
focused  toward  creating  models.  Mathematical  concepts  and  relationships  are  frequently  communicated 
with  mathematical  language,  using  symbolic  representations  where  appropriate.  Students  at  the  Advanced 
level  consider  the  reasonableness  of  an  answer,  with  both  number  sense  and  geometric  awareness.  Their 
abstract  thinking  ability  allows  them  to  create  unique  problem-solving  techniques  and  explain  the  reasoning 
processes  they  followed  in  reaching  a conclusion.  These  students  can  probe  through  examples  and 
counterexamples  that  allow  generalization  and  description  of  assumptions  with  models  and  elegant 
mathematical  language. 


Figure  F-2  (continued) 

Draft  Descriptions  of  the  Achievement  Levels 
Prepared  by  the  Original  Level-setting  Panel 


Twelfth-Grade  Draft  Descriptions 

BASIC.  This  level  represents  understanding  of  fundamental  algebraic  operations  with  real 
numbers,  including  the  ability  to  solve  two-step  computational  problems.  It  also  signifies  an  understanding 
of  elementary  geometrical  concepts  such  as  area,  perimeter,  and  volume,  and  the  ability  to  make 
measurements  of  length,  weight,  capacity,  and  time.  Also  included  at  the  Basic  level  is  the  ability  to 
comprehend  data  in  both  tabular  and  graphical  form  and  to  translate  between  verbal,  algebraic,  and 
graphical  forms  of  linear  expression.  Students  at  this  level  should  be  able  to  use  a calculator  appropriately. 


PROFICIENT.  This  level  represents  mastery  of  fundamental  algebraic  operations  and  concepts 
with  real  numbers,  and  an  understanding  of  complex  numbers.  It  also  represents  understanding  of 
polynomials  and  their  graphs  up  to  the  second  degree,  including  conic  sections.  The  elements  of  plane, 
solid,  and  coordinate  geometry  should  be  understood  at  the  Proficient  level.  The  Proficient  level  includes 
the  ability  to  apply  concepts  and  formulas  to  problem  solving.  Students  at  this  level  should  demonstrate 
critical  thinking  skills.  The  Proficient  level  also  represents  the  ability  to  judge  the  reasonableness  of  answers 
and  the  ability  to  analyze  and  interpret  data  in  both  tabular  and  graphical  form.  Basic  algebraic  concepts, 
measurement,  and  constructive  geometry  concepts  are  mastered  at  this  level. 


ADVANCED.  The  Advanced  level  represents  mastery  of  trigonometric,  exponential,  logarithmic, 
and  composite  functions,  zeros  and  inverses  of  functions,  polynomials  of  the  third  degree  and  higher, 
rational  functions,  and  graphs  of  all  of  these.  In  addition,  the  Advanced  level  represents  mastery  of  topics  in 
discrete  mathematics  including  matrices  and  determinants,  sequences  and  series,  and  probability  and 
statistics,  as  well  as  topics  in  analytic  geometry.  The  Advanced  level  also  signifies  the  ability  to  successfully 
apply  these  concepts  to  a variety  of  problem-solving  situations. 
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Figure  F-3 

Revised  Draft  Descriptions  of  the  Achievement  Levels 
Recommended  by  the  Follow-Up  Validation  Panel 


Revised  Fourth-Grade  Draft  Descriptions 

BASIC.  Basic  students  exhibit  some  evidence  of  conceptual  and  procedure  understanding  in  the 
five  NAEP  content  strands.  At  the  fourth  grade  level,  algebra  and  functions  are  treated  in  informal  and 
exploratory  ways  often  through  the  study  of  patterns.  Basic  students  estimate  and  use  basic  facts  to  perform 
simple  computations  with  whole  numbers.  These  students  show  some  understanding  of  fractions  and 
decimals.  They  solve  simple  real  world  problems  in  all  areas.  These  students  use,  although  not  always 
accurately,  four-function  calculators,  rulers,  and  geometric  shapes.  Written  responses  are  often  minimal  and 
lack  supporting  information. 


PROFICIENT.  Proficient  students  consistently  integrate  procedural  knowledge  and  conceptual 
understanding  as  applied  to  problem  solving  in  the  five  NAEP  content  strands.  Using  whole  numbers  they 
estimate,  compute,  and  determine  whether  their  results  are  reasonable.  They  have  a conceptual 
understanding  of  fractions  and  decimals.  Solving  real  world  problems  in  all  areas  is  important  at  this  level. 
Proficient  students  appropriately  use  four-function  calculators,  rulers  and  geometric  shapes.  These  students 
use  problem  solving  strategies  such  as  identifying  and  using  appropriate  information.  [Problem-solving 
strategies  include  identification  and  use  of  appropriate  information.]  They  present  organized  written 
solutions  with  supporting  information  and  explain  how  they  were  achieved. 


ADVANCED.  Advanced  students  integrate  procedural  knowledge  and  conceptual  understanding  as 
applied  to  problem  solving  in  the  five  NAEP  content  strands.  They  solve  complex  and  nonroutine  real-world 
problems  in  all  areas.  They  have  mastered  the  use  of  tools  such  as  four-function  calculators,  rulers  and 
geometric  shapes.  Advanced  students  draw  logical  conclusions  and  justify  answers  and  solution  processes 
by  explaining  the  “why”  as  well  as  the  “how.”  Interpretations  extend  beyond  obvious  connections  and 
thoughts  are  communicated  clearly  and  concisely. 


Revised  Eighth-Grade  Draft  Descriptions 

BASIC.  Basic  students  exhibit  evidence  of  conceptual  and  procedural  understanding.  These 
students  compare  and  contrast,  find  patterns,  reason  from  graphs,  and  understand  spatial  reasoning.  This 
level  of  performance  signifies  an  understanding  of  arithmetic  operations,  including  estimation,  on  whole 
numbers,  decimals,  fractions,  and  percents.  Students  complete  problems  correctly  with  the  help  of  structural 
prompts  such  as  diagrams,  charts,  and  graphs.  As  students  approach  the  proficient  level,  they  will  solve 
problems  involving  the  use  of  available  data,  and  determine  what  is  necessary  and  sufficient  for  a correct 
solution.  Students  use  problem  solving  strategies  and  select  appropriate  tools,  including  calculators, 
computers,  and  manipulatives  (geometric  shapes)  to  solve  problems  from  the  five  content  strands.  Students 
use  fundamental  algebraic  and  informal  geometric  concepts  to  solve  problems.  Students  at  this  level 
demonstrate  limited  skills  in  communicating  mathematically. 
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PROFICIENT.  Profit  lent  students  apply  mathematical  concepts  and  procedures  consistently  to 
complex  problems.  They  make  conjectures,  defend  their  ideas,  and  give  supporting  examples.  They  have 
developed  the  ability  to  relate  the  connections  between  fractions,  percents,  and  decimals,  as  well  as  other 
mathematical  topics,  such  as  algebra  and  functions.  The  proficient  level  denotes  a thorough  understanding 
of  the  arithmetic  operations  listed  at  the  basic  level.  This  understanding  is  sufficient  to  permit  applications 
to  problem  solving  in  practical  situations.  Quantity  and  spatial  relationships  are  familiar  situations  for 
problem  solving  and  reasoning,  and  students  at  this  level  convey  the  underlying  reasoning  skills  beyond  the 
level  of  arithmetic.  Proficient  students  compare  and  contrast  mathematical  ideas  and  generate  their  own 
examples.  These  students  make  inferences  from  data  and  graphs;  they  understand  the  process  of  gathering 
and  organizing  data,  calculating,  evaluating,  and  communicating  the  results  within  the  domain  of  statistics 
and  probability.  Proficient  students  apply  the  properties  of  informal  geometry,  and  accurately  use  the 
appropriate  tools  of  technology. 


ADVANCED.  Advanced  students  go  beyond  recognition,  identification,  and  application  of 
mathematical  rules  in  order  to  generalize  and  synthesize  concepts  and  principles.  Generalization  often  takes 
shape  through  probing  examples  and  counter  examples  and  can  be  used  to  create  models.  Mathematical 
concepts  and  relationships  are  frequently  communicated  with  mathematical  language,  using  symbolic 
representations  where  appropriate.  Students  at  the  advanced  level  consider  the  reasonableness  of  an  answer, 
with  both  number  sense  and  geometric  awareness.  Their  abstract  thinking  allows  them  to  create  unique 
problem  solving  techniques  and  explain  the  reasoning  processes  they  followed  in  reaching  a conclusion. 
These  students  probe  examples  and  counter  examples  that  allow  generalization  and  description  of 
assumptions  with  models  and  elegant  mathematical  language. 


Revised  Twelfth-Grade  Draft  Descriptions 

BASIC.  Basic  students  demonstrate  procedural  and  conceptual  knowledge  in  solving  problems  in 
the  five  NAEP  content  strands.  They  use  estimation  to  verify  solutions  and  determine  the  reasonableness  of 
the  results  to  real  world  problems.  Algebraic  and  geometric  reasoning  strategies  are  used  to  solve  problems. 
These  students  recognize  relationships  in  verbal,  algebraic,  tabular,  and  graphical  forms.  Basic  students 
demonstrate  knowledge  of  geometric  relationships  as  well  as  corresponding  measurement  skills.  Statistical 
reasoning  is  applied  to  the  organization  and  display  of  data  and  to  reading  tables  and  graphs.  These  students 
generalize  from  patterns  and  examples  in  the  areas  of  algebra,  geometry,  and  statistics.  They  communicate 
mathematical  relationships  and  reasoning  processes  with  correct  mathematical  language  and  symbolic 
representations.  Calculators  are  used  appropriately  to  solve  problems. 

PROFICIENT.  Proficient  students  integrate  mathematical  concepts  and  procedures  consistently  to 
more  complex  problems  in  the  five  NAEP  content  strands.  They  demonstrate  an  understanding  of  algebraic 
reasoning,  geometric  and  spatial  reasoning,  and  statistical  reasoning  as  applied  to  other  areas  of 
mathematics.  They  perform  algebraic  operations  involving  polynomials,  justify  geometric  relationships,  and 
judge  and  defend  the  reasonableness  of  answers  in  real  world  situations.  These  students  analyze  and 
interpret  data  in  tabular  and  graphical  form.  Proficient  students  understand  and  use  elements  of  the  function 
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concept  in  symbolic,  graphical  and  tabular  form.  They  make  conjectures,  defend  their  ideas,  and  give 
supporting  examples. 

ADVANCED.  Advanced  students  consistently  demonstrate  the  integration  of  procedural  and 
conceptual  knowledge,  as  well  as  the  synthesis  of  ideas,  in  the  five  NAEP  content  strands.  Advanced 
students  understand  the  function  concept,  and  they  compare  and  apply  the  numeric,  algebraic,  and  graphical 
properties  of  functions.  They  apply  and  connect  their  knowledge  of  algebra,  geometry,  and  statistics  to  solve 
problems  in  more  advanced  areas  of  continuous  and  discrete  mathematics.  Advanced  students  formulate 
generalizations  using  examples  and  counter  examples  to  create  models.  In  communicating  their 
mathematical  reasoning,  these  students  demonstrate  clear,  concise,  and  correct  use  of  mathematical 
symbolism  and  logical  thinking. 
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Meeting  Participants , NAEP  Mathematics  Achievement  Level-Setting 
Original  Meeting , St.  Louis , Missouri,  March  20-24,  1992 


Marge  Blizard 

Blizard  Professional  Cleaning 
Franklin,  Connecticut 

Christopher  Chomyak 
The  Episcopal  Church 
Calais,  Maine 
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Met  Life 

Crownsville,  Maryland 
Mary  Norman 

Dekalb  County  Board  of  Education 
Decatur,  Georgia 

Janice  Wamsley 
Alcom  School  System 
Glen,  Mississippi 

Ronald  Higgins 

Walla  Walla  School  District 

Walla  Walla,  Washington 

Leona  Lee 

Baltimore  City  Public  Schools 
Baltimore,  Maryland 

Lisa  Bietau 

USD  383  Manhattan  Public  Schools 
Manhattan,  Kansas 

Marsha  Davis 
Alcom  County  Schools 
Corinth,  Mississippi 

Jean  Bush  Ragin 
Patterson  High  School 
Baltimore,  Maryland 


Bill  Oldham 
Harding  University 
Searcy,  Arkansas 
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Retired  Principal 
Draper,  Utah 

Marsha  Stovey 
Detroit  Public  Schools 
Detroit,  Michigan 

Vance  Morris 

Dekalb  County  Board  of  Education 
Atlanta,  Georgia 

Carol  Ballentine 
Duval  County  Schools 
Jacksonville,  Florida 

Tami  Harvey,  ESD 
Audiometric  Technician 
Bums,  Oregon 

Laurence  Payne 
Greater  Houston  Coalition 
for  Educational  Excellence 
Houston,  Texas 

Cheryl  Yunk 
USD  383 

Manhattan,  Kansas 

Kirby  Gchachu 

Zuni  Public  School  District 

Zuni,  New  Mexico 

Corliss  Hubert 

Rutherford  Board  of  Education 
Englewood,  New  Jersey 
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Alcorn  County  Schools 

Woodstock  Union  High  School 

Corinth,  Mississippi 
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Gloria  Moran 
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Williams  Junior  High 

Parkland  College 

Bridgewater,  Massachusetts 

Champaign,  Illinois 

Charles  Jackson 

Joanne  Greaver 

Blairsville,  Pennsylvania 
Cassandra  Turner 

Jefferson  County  Public  Schools 
Louisville,  Kentucky 

Internal  Revenue  Service 

Ellie  Cucinatto 

Miami,  Florida 

Bridgewater  Public  Schools 

Jack  Deal 

Bethel  Park  School  District 

Bridgewater,  Massachusetts 

Pittsburgh,  Pennsylvania 

Lillie  Carr 

Pender  County  Schools 

Ninfa  Rivera 
Lyford  CISD 

Teachey,  North  Carolina 

Raymondville,  Texas 

Eric  Cain 
IBM 

Gerald  Zeringue 

Garrity  Construction  Company 

Metairie,  Louisiana 

Harvey,  Louisiana 

Phillip  Stroup 
Butler  County  MR/DD 

Linda  Brown 

Van  Zile  Elementary  School 

Seven  Mile,  Ohio 

Detroit,  Michigan 

Mike  Gobel 

Walla  Walla  School  District 

Judy  Bibb 

Lonoke  High  School 

Walla  Walla,  Washington 

Cabot,  Arkansas 

Juanita  Tietze 
Retired  Principal 

David  Rank 

School  District  of  Greenville 

Canton,  Ohio 

Greenville,  South  Carolina 

Bill  Cramer,  Jr. 

Cramer  & Mallon,  Attorneys  at  Law 

John  Sweeney 
Freed-Haideman  University 
Henderson,  Tennessee 

Bums,  Oregon 

n r\r\ 


Figure  F-4  (continued) 

Meeting  Participants,  NAEP  Mathematics  Achievement  Level-Setting 
Original  Meeting,  St.  Louis,  Missouri,  March  20-24,  1992 


Norma  Newman 

Ysleta  Independent  School  District 
El  Paso,  Texas 

Barbara  Bayne 

Greenville  County  School  District 
Greenville,  South  Carolina 

William  Rickenbach 
Bethel  Park  School  District 
Bethel  Park,  Pennsylvania 

Landa  McLaurin 
Baltimore  City  Schools 
Baltimore,  Maryland 

Violet  Cosgrove 
Retired 

Glen  Bumie,  Maryland 
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St.  Xavier  University 
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Pre-Mc,  Inc. 
Allen,  Oklahoma 
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Baltimore  City  Public  Schools 
Baltimore,  Maryland 

Bill  Anderson 

Administration  Eagle  Union 
Zionsville,  Indiana 

Jeane  Joyner 

Dept,  of  Public  Instruction 
Raleigh,  North  Carolina 

Dan  Thompson 

Thompson  Construction  Company 
Trinidad,  Colorado 

Carolyn  Craig 

State  Department  of  Education 
Jackson,  Mississippi 

Nancy  Gallagher 

West  Penn  Power  Company 

Kittanning,  Pennsylvania 

Mary  Bennion 
SFUSD 

San  Francisco,  California 

William  Hawes 
The  Hawes  Company 
Tucker,  Georgia 

Bill  Eyestone 

Walla  Walla  School  District 
Walla  Walla,  Washington 

Zhining  Qin 

Minnesota  Department  of  Education 
St.  Paul,  Minnesota 

Mary  Gahn 

New  York  Board  of  Education 
Westbury,  New  York 

Charles  McGee 

Greenville  County  School  District 
Greenville,  South  Carolina 

Barbara  Faltz- Jackson 
Baltimore  City  Public  Schools 
Baltimore,  Maryland 
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Philip  Brach 
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Washington,  D.C. 
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Slippery  Rock  University 
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Consultant  (Self-Employed) 
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Santa  Ana  Unified 
Fountain  Valley,  California 


Ricardo  Suarez 
Lyford  CISD 
Raymondville,  Texas 
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Meeting  Participants , NAEP  Mathematics  Achievement  Level-Setting 
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Michigan  Department  of  Education 
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Largo  Public  Schools 
Largo,  Florida 
Henry  Kepner 
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CORRECTION  OF  THE  NAEP  PROGRAM  DOCUMENTATION  ERROR 
IN  THE  1992  STATE  MATHEMATICS  RESULTS 

Frank  Jenkins  and  Edward  Kulick 
Educational  Testing  Service 


In  April  1995,  results  from  the  1994  Trial  State  Assessment  in  reading  were  released  as  part  of 
the  report  1994  NAEP  Reading:  A First  Look  (Williams,  Reese,  Campbell,.  Mazzeo,  & Phillips,  1995). 
Subsequently,  ETS/NAEP  research  scientists  discovered  an  error  in  the  documentation  for  the  ETS 
version  of  the  PARSCALE  program,  which  is  used  to  compute  NAEP  scale  score  results.  The  error 
affected  how  omitted  responses  were  treated  in  the  IRT  scaling  of  the  extended  constructed-response 
items  that  received  partial-credit  scoring  (i.e.,  could  have  several  partially  correct  categories)  in  analyses 
of  1992  and  1994  assessment  data.  The  error  affected  only  polytomous  items;  omitted  multiple-choice 
and  omitted  short  constructed  responses  were  treated  appropriately. 

The  conventional  treatment  in  NAEP  subjects  has  been  to  treat  omitted  responses  (blank 
responses  to  an  item  that  are  followed  by  valid  responses  to  items  that  appear  later  in  the  test)  as  the 
lowest  possible  score  category  in  the  production  of  NAEP  scale  scores.  In  contrast,  not-reached 
responses  (blank  responses  that  are  not  followed  by  any  further  student  responses)  are  treated  as  missing 
data.  As  a result  of  the  documentation  error,  for  a number  of  the  partial  credit  (or  polytomous) 
constructed-response  items  and  across  several  subject  areas,  all  blank  responses  (both  omitted  and  not- 
reached  responses)  to  affected  items  were  treated  as  missing — a reasonable  model  for  treating  omits  but 
one  that  does  not  conform  to  the  conventional  practice  in  NAEP. 

The  error  occurred  because  of  a documentation  error  in  the  description  of  one  of  the  PARSCALE 
control  parameters,  designated  as  POMIT.  The  program  permits  the  analyst  to  choose  two  different  ways 
of  treating  blank  responses  for  partial  credit  items:  (a)  as  missing  data,  and  (b)  as  incorrect,  i.e.  a valid 
response  falling  in  the  lowest  score  category.  The  documentation  indicates  that  by  setting  POMIT=  -1, 
the  treatment  in  (a)  occurs.  By  setting  POMIT=0  or  POMIT=l,  the  treatment  in  (b)  is  supposed  to  occur. 
The  POMIT=l  setting  is  the  program  default.  In  reality,  P0M1T=1  and  POMIT=  -1  operate  equivalently, 
treating  blank  responses  as  missing  data. 

The  error  appears  to  have  been  introduced  in  1992  when  the  programs  BILOG  and  PARSCALE 
were  merged  o form  the  ETS  version  of  PARSCALE.  Verification  of  the  accuracy  of  existing 
documentation,  modifications  to  internal  program  diagnostics,  and  more  systematic  testing  procedures 
for  any  and  all  changes  to  NAEP-related  programs  were  implemented  immediately  to  reduce  the 
likelihood  of  experiencing  this  kind  of  error  in  subsequent  NAEP  cycles. 

The  PARSCALE  documentation  error  affected  a number  of  the  NAEP  scales  constructed  since 
1992.  Specifically,  the  1992  national  and  state  mathematics  results  were  affected  by  the  error.  Results 
from  these  two  assessments  have  been  released  to  the  public  in  a number  of  NAEP  publications.  The  data 
has  also  been  available  to  the  public  through  NCES’s  secondary-use  data  files. 
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NCES  and  ETS  felt  that  the  most  technically  correct  plan  of  action  would  be  to  recalculate  all 
affected  NAEP  scales,  no  matter  how  slight  the  change,  and  to  issue  revised  results.  ETS  was  therefore 
instructed  by  NCES  to  recalculate  all  affected  scales. 

In  recomputing  the  cutpoints  for  the  achievement  levels,  an  additional  error  (the  information 
weighting  error)  was  discovered  in  the  procedures  used  by  American  College  Testing  (ACT)  in  1992  to 
“map”  the  achievement-level  cutpoints  onto  the  NAEP  scale.  The  procedures  contained  an  incorrectly 
derived  formula.  Details  can  be  found  in  Appendix  I of  the  Technical  Report  of  the  NAEP  1994  Trial 
State  Assessment  in  Reading  (Mazzeo,  Allen,  & Kline,  1995).  ACT  used  revised  procedures  with  the 
correct  formula  to  map  the  achievement-level  cutpoints  for  the  1994  U.S.  history  and  geography  scales 
and  the  1996  science  scales.  The  error  in  the  procedures  affected  achievement-level  cutpoints  for  the 
1990  mathematics  national  assessment  at  all  grade  levels  (grades  4,  8,  and  12)  and  the  1990  Trial  State 
Assessment  in  mathematics,  which  was  only  at  grade  8.  This  error  also  affected  the  1992  mathematics 
results  for  the  nation  and  the  states.  The  information  weighting  error  added  a source  of  error  to  results  in 
addition  to  the  error  associated  problem  with  defining  omits.  Note  that  the  proficiency  estimates  for  1990 
mathematics  are  correct  and  did  not  have  to  be  recalculated;  only  the  cutpoints  for  the  achievement 
levels  were  affected.  For  this  reason,  the  achievement  level  almanac  for  1990  state  mathematics  is  not 
included  in  this  appendix. 

The  information  documenting  the  original  analysis  of  the  1992  data  that  appears  in  the  Technical 
Report  of  the  NAEP  1992  Trial  State  Assessment  Program  in  Mathematics  (Johnson,  Mazzeo,  & Kline, 
1993)  is  substantially  in  agreement  with  the  revised  1994  analysis.  The  transformation  constants  for  the 
revised  analysis  are  provided  in  Table  G-l . The  information  in  the  other  sections  of  the  technical  report 
for  the  1996  state  mathematics  assessment  refer  only  to  the  revised  analysis  of  the  1994  Trial  State 
Assessment  data. 


Table  G-l 

Transformation  Constants  for  the  1992  Trial  State  Assessment  in  Mathematics 


Grade  4 

Grade  8 

Scale 

k, 

k 2 

ki 

k2 

Numbers  and  Operations 

214.59 

34.16 

268.76 

34.60 

Measurement 

221.40 

33.28 

262.82 

43.95 

Geometry 

220.55 

28.59 

260.44 

33.81 

Data  Analysis,  Statistics,  and  Probability 

217.80 

32.66 

264.58 

39.93 

Algebra  and  Functions 

217.91 

29.00 

264.23 

36.13 

Estimation 

205.41 

35.52 

267.14 

28.14 

As  shown  by  Tables  G-3  through  G-5,  all  jurisdictions  had  average  scores  that  were  adjusted 
upward  slightly  as  a result  of  the  revision  of  results.  For  grade  4,  Tables  G-3  and  G-4  indicate  that 
average  scores  increased  from  .9  to  1 .5  points  on  the  proficiency  scale.  Accordingly,  average  scores  in  all 
the  percentiles  go  up  in  a similar  fashion.  Since  all  jurisdictions  were  affected  in  a similar  manner,  there 
is  little  change  in  the  ranking  of  the  jurisdictions.  With  regard  to  achievement  level  results,  Tables  G-7 
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and  G-8  indicate  that  although  the  average  scale  scores  moved  up  when  revised,  the  percent  of  students 
at  or  above  the  advanced,  proficient  and  basic  achievement  levels  went  down  slightly  as  a result  of  the 
two  revisions.  For  the  advanced  level,  percentages  for  the  jurisdictions  went  down  0 to  .8  percent,  while 
for  the  below  basic  level,  percentages  went  up  .6  to  1.7  percent.  Since  the  scale  scores  uniformly  moved 
up,  the  shift  in  achievement  level  percentages  must  be  due  to  the  shift  upward  of  the  achievement  level 
cutpoints  as  result  of  the  information  weighting  error  (see  Table  G-2).  Similar  results  are  evident  for 
grade  8. 


Tables  G-5  and  G-6  indicate  that  average  scores  for  jurisdictions  went  up  from  .6  to  1 point  as  a 
result  of  the  revision  of  scale  scores,  with  similar  slight  upward  adjustments  occurring  at  every  listed 
percentile.  This  change  is  somewhat  less  than  what  was  demonstrated  for  grade  4 and  again  the  ranking 
of  the  jurisdictions  did  not  show  much  change.  Tables  G-9  and  G-10  list  the  original  and  revised  eighth- 
grade  results  for  the  achievement-level  percentages.  These  tables  also  list  values  for  1990,  since  the  Trial 
State  Assessment  was  administered  only  at  grade  8 and  not  grade  4 that  year.  The  effects  of  the  revision 
of  the  1992  results  are  similar  to  those  for  grade  4.  Although  revised  means  were  greater  for  every 
jurisdiction,  the  percentage  above  the  three  achievement  levels  were  slightly  smaller  while  percentages 
below  the  basic  achievement  level  were  slightly  larger.  As  with  grade  4,  this  reflects  the  upward  shift  of 
the  achievement-level  cutpoints  as  a result  of  the  information  weighting  error  (see  Table  G-2). 

For  grade  8,  there  is  the  question  of  the  effect  of  the  revisions  on  the  1990  to  1992  trend.  All  of 
the  17  jurisdictions  that  had  significant  trends  in  average  scale  scores  in  the  original  analysis  also  had 
significant  trends  in  the  revised  analysis.  In  addition,  seven  jurisdictions  that  did  not  have  significant 
trends  in  the  original  analysis  had  significant  trends  in  revision.  A number  of  changes  in  the  percent  of 
students  at  or  above  the  achievement  levels  occurred  due  to  the  combination  of  the  revision  of  scale 
scores  and  achievement  level  cutpoints  for  the  1992  data.  As  a result,  more  differences  across  the  two 
years  were  significant.  For  15  jurisdictions,  there  was  an  increase  in  the  percent  of  students  at  or  above 
an  achievement  level  from  1990  to  1992  that  did  not  change  when  the  analysis  was  redone.  There  were 
nine  jurisdictions  where  there  was  a significant  trend  upward  in  the  percent  of  students  at  or  above  an 
achievement  level  only  after  revision.  For  only  one  jurisdiction  was  a trend  no  longer  significant  after 
revision  (the  percent  at  or  above  advanced  for  Michigan). 

Table  G-2 

Original  and  Revised  ACT  Achievement  Level  Cut  Scores  for 
1 992  Mathematics 


Basic 

Proficient 

Advanced 

Grade  4 

Original 

211 

248 

280 

Revised 

214 

249 

282 

Grade  8 

Original 

256 

294 

331 

Revised 

262 

299 

333 

In  summary,  while  changes  in  average  scale  score,  in  percent  of  students  at  or  above  an 
achievement  level,  and  in  1990  to  1992  trends  in  these  statistics  did  occur,  the  meaning  of  the  results 
were  constant  for  most  jurisdictions.  An  exception  was  that  seven  jurisdictions  had  significant 
differences  in  mean  scale  score  from  1990  to  1992  that  did  not  appear  in  the  1992  state  reports.  Also,  10 
jurisdictions  had  changes  in  the  percent  of  students  at  or  above  at  least  one  achievement  level. 


A1  abasia 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District  of  Columbia 

riorida 

Georgia 

Hawaii 

Idaho 

Indiana 

Iowa 

Kentucky 

Louisiana 

Maine 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

Missouri 

Nebraska 

New  Hampshire 
New  Jersey 
New  Mexico 
New  York 
North  Carolina 
North  Dakota 

Ohio 

Oklahoma 
Pennsylvania 
Rhode  Island 
South  Carolina 

Tennessee 

Texas 

Utah 

Virginia 
West  Virginia 
Wisconsin 
Wyoming 

Guam 

Virgin  Islands 


Table  G-3 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  4 Weighted  Percentages  and  Composite  Scale  Means 
Weighted  Means,  Standard  Deviations , and  Percentiles 
Original  Results 


MEAN 


206.94 

1.6) 

213.84 

1.1) 

208.74 

0.9) 

207.14 

1.6) 

219.84 

1.0) 

225.84 

1.2) 

216.64 

0.8) 

191.24 

0.5) 

212.44 

1.5) 

214.34 

1.3) 

212.84 

1.3) 

220.34 

1.0) 

219.7 ( 

1.1) 

229.04 

1.1) 

213.64 

1.0) 

202.84 

1.4) 

230.74 

1.0) 

216.14 

1.3) 

225.54 

1.2) 

218.64 

1.8) 

227.54 

0.9) 

200.14 

1.1) 

221.04 

1.2) 

224.24 

1.3) 

228.64 

1.2) 

226.14 

1.5) 

211.84 

1.5) 

217.24 

1.3) 

211.44 

1.1) 

227.64 

0.8) 

217.54 

1.2) 

219.04 

1.0) 

223.24 

1.4) 

214.04 

1.6) 

211.14 

1.1) 

209.44 

1.4) 

216.64 

1.3) 

222.84 

1.0) 

219.64 

1.3) 

213.94 

1.1) 

227.7  4 

1.1) 

224.24 

1.0) 

191.14 

0.8) 

178.04 

1.2) 

STD  DSTV 

32. 0(  0.6) 
31. 3<  0.5) 
30. 9<  0.6) 
36. 6(  0.8) 
31. 2(  0.4) 

32.14  0.7) 

32. 4(  0.7) 
32. 8(  0.4) 
32. 6<  0.8) 
32. 8(  0.6) 
34. 0(  0.7) 
28. 1{  0.5) 

28. 3(  0.5) 
29. 6{  0.5) 
29. 8(  0.6) 
32. 2(  0.9) 
28. 3(  0.7) 
35. 2(  0.8) 

31. 2(  0.7) 
32. 9(  1.0) 
31. 1(  0.6) 

31.84  0.6) 
30. 3(  0.7) 
30. 8(  0.6) 

28. 7{  0.5) 

31.44  0.9) 
30. 2(  0.7) 
32. 8(  0.9) 

33. 14  0,6) 

26.94  0.6) 

31. 4(  0.8) 
27. 4 ( 0.6) 

31.44  0.7) 
32. 0(  0.9) 
31. 8(  0.6) 
30. 8(  0.6) 

31. 3(  0.8) 
29. 3(  0.6) 
32. 6(  0.7) 
30. 1(  0.6) 
29. 6(  0.7) 
27. 2(  0.5) 

34.34  0.6) 
28. 3(  0.7) 


10TH 

165. 3(  1.7) 
171. 8{  2.1) 
167. 1(  1.3) 
157. 6(  2.4) 
178. 8(  1.7) 

183.64  2.2) 

174. 6(  1.5) 
151. 7(  0.7) 
169. 6(  2.5) 
171. 3{  2.0) 
167. 5(  1.7) 
183. 4(  1.4) 

183.64  1.2) 

190.54  2.2) 

175.64  1.7) 
160 . 4 { 2.3) 
194. 0{  1.9) 
169. 5{  2.0) 

184. 4{  1.5) 
174. 3(  3.3) 
185. 9(  3.8) 
159. 2(  1.7) 
181. 6(  2.9) 

183.34  1.9) 

191.64  1.6) 

184. 64  2.9) 
172. 5(  2.5) 
173. 3(  3.2) 
167. 7(  1.6) 
192. 8(  2.8) 

177. 2(  2.7) 
183. 7(  1.5) 
181. 0<  1.8) 
171. 9<  3.1) 
170. 7(  1.3) 

169.34  2.0) 

176. 6(  2.2) 
184. 8<  1.6) 
177. 5(  1.4) 
174. 9(  1.5) 
188. 5(  2.1) 
189. 1(  2.1) 

147. 1(  1.8) 
140. 4(  1.8) 


25TH 

184. 2(  1.5) 

192.64  1.7} 
187. 4(  1.1) 
183. 0{  2.7) 
199. 5(  1.0) 
204. 6{  1.6) 

193.74  1.0) 
169. 0(  0.7) 
191. 2(  2.0) 
191. 8(  1.2) 
190. 1(  1.6) 
201. 7(  2.5) 

199.94  1.7) 
210. 0(  1.1) 
193. 3(  1.1) 
181. 3(  1.6) 
212. 2(  1.2) 
191. 5(  2.6) 

205. 3<  1.5) 
197. 9(  2.3) 
207. 7(  1.2) 
178. 3(  1.0) 
201. 2<  1.4) 
203. 8(  1.7) 

210. 0(  1.2) 
205. 6(  1.9) 
191. 1(  1.8) 
195. 9(  1.4) 
188. 2(  1.4) 
210. 2{  1.9) 

196. 4(  1.5) 

200.94  1.2) 
202. 5<  1.9) 
193. 2(  2.9) 
189. 1(  1.2) 
188. 3(  1.9) 

196. 3(  1.5) 
203. 7{  0.9) 
197. 4<  1.6) 
193. 9(  1.4) 
208. 7(  1.0) 
206. 8(  1.9) 

167. 5(  0.9) 

159.24  2.2) 


50TH 

206. 84  2.4) 
215. 4(  1.3) 
210. 0(  1.3) 
209. 3(  1.3) 
221. 0<  1.4) 
227.0<  1*7) 

216.4(  1.0) 

189.54  0.7) 

213.64  1.8) 

215.24  1.3) 

214.14  1.9) 

221.94  0.9) 

219.74  1.2) 

230.54  0.8) 

213.14  1.1) 

203.14  2.0) 

231.94  1.8) 

217.64  2.2) 

227.04  1.3) 

221.14  1.8) 

229.54  1.0) 

200.24  1.3) 

221.84  1.5) 

225.84  1.1) 

229.24  1.4) 
227.9(  1.6) 

212.34  1.0) 

218.54  1.8) 

212.74  1.4) 

228.34  0.8) 

218.04  1.3) 

219.24  1.1) 

224.84  1.8) 

215.74  1.8) 

210.34  1.2) 

210.34  1.7) 

217.04  1.6) 

223.84  1.3) 

219.84  1.1) 

214.14  1.2) 

229.44  1.2) 

225.14  1.2) 

191.44  1.1) 

178.74  1.3) 


75TH 

230.14  1.7) 

235.64  1-3) 

230.44  1.0) 

232.74  1.5) 

241.04  1.4) 

248.24  1.7) 

239.44  1.5) 

210.94  0.9) 

234.54  1.4) 

237.34  1.7) 

236.84  1.2) 

240.04  0.7) 

239.34  1.2) 

249.44  0.7) 

233.94  0.9) 

225.14  3.2) 

250.54  1.0) 

241.14  1.2) 

247.14  1.4) 

241.84  1.5) 

249.44  0.9) 

222.44  1.2) 

242.14  1.2) 

245.54  1.4) 

248.64  1.3) 

248.34  1.1) 

232.24  1.7) 

240.14  1.7) 

234.64  1.3) 

246.34  0.8) 

239.14  2.3) 

237.24  1.4) 

245.74  1.6) 

235.94  2.0) 

233.34  1.4) 

230.84  1.5) 

237.94  1.6) 

242.84  0.8) 

241.94  1.8) 

234.34  1.4) 

248.34  1.3) 

242.94  1-3) 

213.84  1.4) 

197.24  2.2) 


90TH 

248.84  1.8) 

253.34  2.3) 

247.94  1.5) 

252.94  1.9) 

259.24  1.4) 

265.84  1.3) 

258.94  1.4) 

233.24  2.2) 

253.64  2.6) 

256.84  2.0) 

256.34  2.1) 

255.64  1.2) 

256.24  1.0) 

265.84  1.1) 

252.64  1.7) 

244.04  1.5) 

265.14  1.3) 

260.54  1.6) 

264.04  1.1) 

258.94  1.6) 

265.84  1.0) 

240.74  1.8) 

259.74  1.4) 

262.44  1.6) 

264.64  2.5) 

265.04  2.3) 

250.74  2.5) 

258.44  1.6) 

253.04  1.2) 

261.04  1.1) 

257.74  1.4) 

253.84  2.2) 

262.04  2.3) 

253.74  2.2) 

252.94  2.4) 

248.54  1.9) 

256.54  2.7) 

259.84  0.9) 

261.74  2.8) 

252.24  1.8) 

264.24  1.3) 

258.24  1-2) 

235.34  1.3) 

213.04  2.4) 


1 HA 


Alabama 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District  of  Columbia 

Florida 

Georgia 

Hawaii 

Idaho 

Indiana 

Iowa 

Kentucky 

Louisiana 

Maine 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

Missouri 

Nebraska 

New  Hampshire 
New  Jersey 
New  Mexico 
New  York 
North  Carolina 
North  Dakota 

Ohio 
Oklahoma 
Pennsylvania 
Rhode  Island 
South  Carolina 
Tennessee 

Texas 

Utah 

Virginia 

West  Virginia 

Wisconsin 

Wyoming 

Quam 

Virgin  Islands 


Table  G-4 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  4 Weighted  Percentages  and  Composite  Scale  Means 
Weighted  Means , Standard  Deviations,  and  Percentiles 
Revised  Results 


MXAN  STD  DKV 


10TH 


25TH  50TH 


75TH 


90TH 


208. 3( 

1.6) 

215. 3( 

1.1) 

210. 2( 

0.9) 

208. 4( 

1.6) 

221. 0( 

1.0) 

226. 8( 

1.1) 

217. 9( 

0.8) 

192. 6( 

0.5) 

213. 7 ( 

1.5) 

215. 6( 

1.2) 

214. 1< 

1.3) 

221. 6{ 

1.0) 

221. 0( 

1.0) 

229. 9{ 

1.0) 

215. 0( 

1.0) 

204. 1( 

1.5) 

231. 6< 

1.0) 

217. 3( 

1.3) 

226. 6( 

1.2) 

219. 9( 

1.7) 

228. 5( 

0.9) 

201. 8( 

1.1) 

222. 2( 

1.2) 

225. 3( 

1.2) 

229. 7< 

1.2) 

227. 1( 

1.5) 

213. 3( 

1.4) 

218. 4{ 

1.2) 

212. 9< 

1.1) 

228. 7{ 

0.8) 

218. 7< 

1.2) 

220. 3( 

1.0) 

224. 3( 

1.3) 

215.4 ( 

1.5) 

212. 5( 

1.1) 

210. 9< 

1.4) 

217. 9 ( 

1.2) 

224. 0( 

1.0) 

220. 8( 

1.3) 

215. 3( 

1.1) 

228. 7( 

1.1) 

225. 4< 

0.9) 

192.8 ( 

0.8) 

178. 9( 

1.2) 

31. 3( 

0.6) 

30. 3( 

0.5) 

30. 1( 

0.6) 

36. 0( 

0.8) 

30. 3( 

0.4) 

31. 0( 

0.7) 

31. 4{ 

0.7) 

32. 6( 

0.4) 

31. 8< 

0.7) 

31. 9{ 

0.6) 

33. 2( 

0.6) 

27. 2( 

0.5) 

27. 4( 

0.5) 

28. 6( 

0.5) 

28. 9( 

0.6) 

31. 8( 

1.1) 

27. 4< 

0.7) 

34. 3( 

0.8) 

30. 2( 

0.6) 

31. 9< 

1.0) 

30. 0( 

0.5) 

31. 1( 

0.6) 

29. 3( 

0.7) 

29. 7( 

0.6) 

27 .6  { 

0.5) 

30. 3( 

0.9) 

29. 3< 

0.6) 

32. 0( 

0.9) 

32. 2( 

0.6) 

25. 9{ 

0.5) 

30. 5( 

0.8) 

26. 5( 

0.6) 

30. 4{ 

0.7) 

31. 0( 

0.9) 

31. 0( 

0.6) 

29. 9( 

0.6) 

30. 3( 

0.8) 

28 . 3 ( 

0.6) 

31. 7( 

0.7) 

29. 3{ 

0.5) 

28. 5( 

0.7) 

26.2  ( 

0.5) 

33. 7( 

0.6) 

28. 8{ 

0.7) 

167. 7{  1.4) 
174. 4(  1.8) 
169. 7(  1.5) 
159. 0(  1.9) 
181. 2 ( 1.8) 
185. 8(  2.2) 

177. 0(  1.4) 
153. 2 ( 0.8) 
171. 7(  2.1) 
173. 7(  1.7) 
170. 0(  1.5) 
185. 6(  1.9) 

186. 0(  1.3) 
192. 7(  2.3) 
178. 2(  1.5) 
162. 9<  2.5} 
196. 2(  1.9) 
171. 8<  1.7) 

186. 7{  1.2) 
176. 8(  3.1) 
188. 3(  4.1) 
161. 7(  1.5) 
184. 0{  1.7) 
185. 7{  1.8) 

193. 8<  1.1) 
186. 9(  3.1) 
175. 0(  2.9) 
175. 7{  2.2) 
170. 4(  1.1) 
195. 0(  2.4) 

179. 5(  1.8) 
186. 0<  1.7) 
183. 4(  1.9) 
174. 4<  2.8) 
173. 0(  1.3) 
171. 8(  2.3) 

179. 0{  2.4) 
187. 0(  1.6) 
179. 8(  1.3) 
177. 3{  1.4) 
190. 8(  1.9) 
191. 3(  1.6) 

149. 6<  1.4) 
140. 7(  2.3) 


186. 2{  1.8) 
194. 6(  2.3) 
189. 5(  1.1) 
184. 6(  2.3) 
201. 4(  1.0) 
206. 4(  1.3) 

195. 7(  1.3) 
170. 7<  1.1) 
193. 0(  1.8) 
193. 7(  1.4) 
191. 8{  2.3) 
203. 5(  1.9) 

201. 7(  1.3) 
211. 8(  1.4) 
195. 2(  1.0) 
183. 3(  2.0) 
213. 8(  1.3) 
193. 4{  2.2) 

207. 2(  1.5) 
199. 9(  2.8) 
209. 4(  1.1) 
180. 5<  1.4) 
203. 1(  1.4) 
205. 7(  1.8) 

211. 7(  1.2) 
207. 4(  1.7) 
193. 2(  2.1) 
197. 6(  1.0) 
190. 4<  1.4) 
212. 0(  1.1) 

198. 2(  1.5) 
202. 8(  0.9) 
204. 3{  1.5) 
195. 3<  3.0) 
191. 2{  1.1) 
190. 5(  2.1) 

198. 3(  1.4) 
205. 6(  1.2) 
199. 3(  1.5) 
195. 8<  1.6) 
210. 4(  0.7) 
208. 6(  1.4) 

169. 6(  1.2) 
159. 9(  2.2) 


208. 4( 

2.2) 

216. 9( 

1.3) 

211. 5< 

1.2) 

210. 7( 

1.5) 

222. 3( 

0.9) 

228. 3( 

1.5) 

217. 9 { 

0.8) 

191. 0( 

0.6) 

215. 2< 

1.5) 

216. 7{ 

1.4) 

215. 5( 

1.3) 

223. 3( 

1.2) 

221. 2< 

0.9) 

231. 6< 

0.9) 

214. 7( 

1.0) 

204. 8< 

1.4) 

232. 8( 

1.3) 

219. 0( 

1.5) 

228. 3( 

1.4) 

222. 5< 

1.7) 

230. 6( 

0.9) 

202. 1< 

1.6) 

223. 1( 

1.4) 

227. 0( 

1.3) 

230. 3( 

1.3) 

229. 1( 

1.5) 

213. 9( 

1.2) 

220. 0( 

1.7) 

214. 3< 

1.3) 

229. 4( 

0.8) 

219. 4 < 

0.9) 

220. 6( 

1.2) 

226. 0< 

1.7) 

217.3 ( 

1.9) 

211. 9< 

1.2) 

212. 0( 

1.6) 

218. 5( 

1.5) 

225. 1( 

1.3) 

221. 2( 

1.4) 

215. 6( 

1.1) 

230. 5( 

1.3) 

226. 4( 

1.2) 

193. 2 ( 

1.1) 

179. 8( 

1.7) 

231. 1{ 

1.5) 

236. 5( 

0.9) 

231. 4( 

1.1) 

233. 5< 

1.3) 

241. 7( 

1.2) 

248.5 { 

0.9) 

240. 1( 

1.4) 

211. 8< 

1.2) 

235. 3{ 

1.4) 

238. 2( 

1.4) 

237. 5 ( 

1.1) 

240. 7( 

0.7) 

240. 1( 

1.2) 

249. 7( 

0.7) 

234. 9( 

1.0) 

226. 4( 

2.4) 

250. 8( 

1.0) 

242. 0< 

1.7) 

247.6 ( 

1.6) 

242. 5 ( 

1.6) 

249. 6( 

1.1) 

223. 8( 

1.2) 

242 . 7 ( 

1.1) 

246. 0< 

1.3) 

248. 9( 

1.3) 

248.7 ( 

1.0) 

233. 3( 

2.1) 

240. 8( 

1.6) 

235. 5< 

1.6) 

246. 7{ 

0.8) 

239. 8( 

1.7) 

238. 0( 

1.4) 

246. 1( 

1.8) 

236. 8( 

1.7) 

234 . 2 ( 

1.4) 

231. 9< 

1.4) 

238. 7( 

1.8) 

243. 3 ( 

1.0) 

242. 5{ 

1.9) 

235. 2( 

1.4) 

248. 6{ 

1.2) 

243. 5{ 

0.9) 

215. 4( 

1.3) 

198. 3( 

1.2) 

249. 3( 

2.4) 

253. 4( 

1.4) 

248. 5( 

1.5) 

253. 1< 

2.6) 

259. 1< 

0.8) 

265. 3( 

1.6) 

258. 8( 

1.4) 

234. 4{ 

1.4) 

253. 8( 

3.1) 

256. 8( 

1.2) 

256. 5< 

1.9) 

255. 5( 

1.1) 

256. 3{ 

1.0) 

265. 4< 

1.5) 

252. 8( 

1.3) 

244. 5( 

1.4) 

264. 6 ( 

1.2) 

260. 5< 

1.2) 

263 . 6 ( 

1.7) 

258. 7( 

1.6) 

265. 3( 

1.4) 

241. 6( 

1.2) 

259. 5( 

1.6) 

262. 1( 

1.8) 

264. 1< 

2.3) 

264  6 ( 

1.9) 

250. 9( 

1.8) 

258. 3( 

1.0) 

253. 0( 

1.2) 

260. 8< 

1.4) 

257. 4 ( 

1.1) 

253. 9( 

1.9) 

261. 7( 

1.8) 

253. 8( 

3.0) 

253. 1{ 

1.8) 

248. 8{ 

1.4) 

256. 5( 

2.3) 

259. 8( 

1.0) 

261. 3( 

2.2) 

252. 3( 

1.6) 

263. 6( 

1.8) 

258. 2( 

0.9) 

236. 2( 

1.7) 

215. 0( 

2.5) 

BEST  COPY  AVAILABLE 


5d 


407 


Alabama 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

District  of  Columbia 

Florida 

Georgia 

Hawaii 

Idaho 

Indiana 

Iowa 

Kentucky 

Louisiana 

Maine 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

Missouri 

Nebraska 

New  Hampshire 
New  Jersey 
New  Mexico 
New  York 
North  Carolina 
North  Dakota 

Ohio 

Oklahoma 
Pennsylvania 
Rhode  Island 
South  Carolina 

Tennessee 

Texas 

Utah 
Virginia 
West  Virginia 
Wisconsin 
Wyoming 

Guam 

Virgin  Islands 


Table  G-5 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  8 Weighted  Percentages  and  Composite  Scale  Means 
Weighted  Means,  Standard  Deviations,  and  Percentiles 
Original  Results 


KZAN 

251. 3(  1.7) 
264. 6(  1.3) 
255. 4<  1.2) 
260. 1(  1.7) 
271. 7(  1.1) 
273. 1(  1.1) 

262. 1(  1.0) 
233. 9(  0.9) 
259. It  1.5) 
258 . 5 ( 1.2) 
256. 6(  0.9) 
274. 4(  0.8) 

269. 4<  1.2) 
282. 8(  1.0) 
261. 4(  1.1) 
249. 1(  1.7) 
278. 0<  1.0) 
264. 2(  1.3) 

272. 1(  1.1) 
266. 6(  1.4} 
281. 8(  1.0) 
245. 5(  1.2) 
270. 4(  1.2) 
277. 0(  1.1) 

277. 6(  1.0) 
271. 2<  1.6) 
258. S(  0.9) 
265. 7(  2.1) 
257. 6<  1.2) 
282. 6(  1.2) 

267 .4 ( 1.5) 
267. 4(  1.2} 
270. 7<  1.5) 
265. 1(  0.7) 
260. 0(  1.0) 
258. 0<  1.4) 

263. 8(  1.3) 
273. 6(  0.7) 
267. 1<  1.2) 
258. 2(  1.0) 
277. 3(  1.5) 
274. 4<  0.9) 

234. 3(  1.0) 

221. 8(  1.1) 


STD  DKV 

35. 8(  1.2) 
32. 8(  0.7) 
34. 3(  0.6) 
38. 7(  1.1) 
33. 1{  0.6) 
36. 0(  0.9) 

35. 7(  0.7) 
36. 5{  1.0) 
36. 8(  0.8) 
34. 6(  0.6) 
37. 8{  0.7) 
30. 5{  0.5) 

33. 9(  0.6) 
30. 0(  0.6) 
34. 3<  0.6) 
34. 0(  0.9) 
30. 7(  0.8) 
39. 2(  0.9) 

34. 2(  0.7) 
35. 4(  0.6) 
31. 9(  0.5) 
34. 8(  0.6) 
32. 7(  0.7) 
32. 4<  0.6) 

30. 4(  0.7) 
36. 0(  0.9) 
32. 3(  0.7) 
39. 4<  1.3) 
35. 4(  0.8) 
28. 3(  0.6) 

34. 4(  0.9) 
32. 3(  0.6) 
34. 5{  0.9) 
33. 4(  0.5) 
35. 3(  0.7) 
33. 9<  0.7) 

37. 2(  0.7) 
32. 0{  0.8) 
35. 4(  0.7) 
31. 1(  0.6) 
33. 2{  0.7) 
29. 6(  0.5) 

39. 2(  0.8) 
30. 1(  0.6) 


10TH 


205. 5(  1.9) 
222. 1(  1.6) 
210. 9(  1.6) 
208. 8(  2.7) 
227. 6(  1.6) 
224. 3(  2.6) 

216. 4<  1.8) 
188. 6(  1.0) 
210. 3(  3.0) 
213. 8 ( 1.5) 
208. 2(  1.5) 
234. 9(  1.1) 

225. 3(  1.5) 
243. 9<  2.1) 
216. 4{  1.7) 
204. 6(  2.6) 
239. 1{  2.3) 
212. 9(  1.8) 

228. 6<  1.4) 
220. 0(  1.4) 
240. 0(  1.4) 
200. 6(  1.2) 
227. 9(  2.9) 
234. 0{  1.7) 

237. 9(  1.1) 
222. 2(  1.9) 
217. 1(  2.0) 
213. 2{  3.1) 
212. 3(  2.6) 
244. 9(  1.2) 

222. 0{  1.9) 
225. 9(  1.3) 
225. 1(  2.3) 
220. 9(  1.2) 
215. 2(  1.3) 
214. 0(  2.1) 

215. 8(  2.7) 
231. 6(  1.2) 
221. 4{  1.5) 
217. 8 < 1.5) 
233. 1{  2.6) 
236. 5(  1.0) 

183. 8<  2.1) 
183. 1(  1.2) 


25TH 

226. 8<  1.8) 
242. 8<  1.3) 
232. 9<  1.2) 
233. 7(  2.6) 
250. 1(  1.2) 
248. 8(  1.7) 

238. 7(  0.9) 
208. 7(  1.2) 
234. 0(  1.6) 
234. 7(  1.5) 
230. 9(  1.0) 
254. 5(  0.9) 

246. 8{  1.2) 
262. 5(  1.4) 
238. 5<  1.6) 
226. 4(  2.2) 
258. 2(  1.2) 
236. 9(  2.3) 

248. 6(  2.2) 
243. 1<  2.3) 
260. 2(  1.4) 
221. 3(  1.3) 
248. 5<  1.8) 
256. 4(  1.2) 

257. 7(  0.8) 
247. 0(  2.0) 
237. 4(  0.9) 
240. 9(  2.7) 
233. 6(  1.3) 
263. 7(  1.4) 

244. 4(  2.0) 
247. 1<  1.4) 
247. 9(  1.5) 
242. 8(  1.1) 
235. 2(  1.1) 
234. 9(  1.5) 

237. 5<  1.2) 
252. 8<  1.7) 
243. 0(  1.7) 
237. 0{  1.0) 
256. 7(  2.1) 
254. 1(  1.2) 

207. 1(  1.8) 

201. 2{  1.6) 


50TH 


250. 6(  2.0) 
265. 0(  1.9) 
256. 2(  1.2) 
261. 4(  1.8) 
273. 3(  1.1) 
275. 4(  0.8) 

262. 4(  1.3) 
233. 3(  1.8) 
260. 1{  2.0) 
259. 4<  1.3) 
257. 5(  1.6) 
275. 2(  1.2) 

269. 8(  1.3) 
283. 7(  1.1) 
262. 0(  1.0) 
249. 7<  1.6) 
278. 6(  1.1) 
265. 0(  1.3) 

272. 9(  2.0) 
268. 3(  1.6) 
282. 9(  1.3) 
245. 4(  1.2) 
271. 8{  1.4) 
279. 1(  1.4) 

278. 0(  0.9) 
272. 9(  1.9) 
259. 3(  1.0) 
268. 3(  1.8) 
258. 5(  1.2) 
284. 3(  1.0) 

269. 1(  1.6) 
268. 3<  1.1) 
272. 2(  1.4) 
266. 7(  1.2) 
259. 2<  1.2) 
258. 2(  1.6) 

263. 9{  1.9) 
274. 9<  0.8) 
267. 4{  1.7) 
258. 4(  1.7) 
279. 1(  1.5) 
275. 2(  1.2) 

233. 4(  1.6) 

221. 4(  1.2) 


75TH 


275. 7(  1.7) 
286. 9(  1.2) 
278. 8(  1.6) 
287. 9(  1.7) 
295. 0<  1.2) 
299. 2(  1.0) 

286. 6<  1.5) 
257. 2(  2.8) 
284. 8(  1.7) 
282. 9<  2.1) 
282. 8{  1.0) 
295. 7{  0.8) 

292. 7(  1.9) 
303. 9<  1.5) 
284. 6(  1.3) 
272. 1<  2.0) 
299. 0(  1.7) 
292. 4(  1.6) 

296. 8(  1.6) 
291. 7(  2.9) 
304. 3(  1.4) 
270. 0(  1.6) 
292. 8(  1.6) 
299. 7(  1.0) 

298. 5(  1.1) 
296. 8(  2.2) 
280. 9(  1.0) 
292. 7(  1.4) 
282. 0(  1.4) 
302. 0(  1.4) 

291. 6(  1.4) 
290. 1(  1.4) 
295. 0<  1.1) 
288. 5(  1.7) 
284. 6 ( 1.7) 
282. 1(  1.4) 

289. 4(  2.3) 
296. 1(  1.2) 
291. 4(  1.6) 
280. 5(  1.1) 
300. 6{  1.5) 
294. 9(  1.2) 

261. 4(  2.5) 
242. 1(  1.5) 


BEST  COPY  AVAILABLE 


90TH 

298. 8(  2.0) 
306. 5<  1.3) 
298. 6(  1.6) 
308. 8<  2.5) 
313. 3(  1.2) 
318. 4<  1.4) 

307. 0(  1.4) 
280. 2<  1.7) 
306. 5(  2.0) 
303. 1(  1.5) 
304. 7<  1.3) 
313. 0(  1.1) 

313. 1(  2.9) 
320. 6(  1.6) 
305. 4(  2.8) 
293. 0(  1.8) 
316. 4<  1.3) 
313. 5(  1.6) 

315. 7(  1.7) 
311. 2(  2.3) 
322. 3(  1.4) 
290. 9(  2.0) 
311. 5<  1.3) 
316. 9<  1.5) 

315. 9(  2.0) 
317. 1(  1.6) 
300. 2(  1.3) 
314. 5<  2.4) 
302. 7(  1.5) 
318. 0(  1.7) 

310. 4(  1.5) 
307. 7(  1.5) 
314. 5(  1.8) 
306. 9<  1.1) 
306. 7(  1.5) 
301. 6(  1.5) 

312. 0(  1.5) 
313. 7{  1.2) 
313. 4(  1.5) 
298. 2(  1.8) 
317. 8(  1.4) 
312. 1(  1.1) 

286. 2(  3.7) 
260. 2(  1.6) 


ADS 


Table  G-6 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  8 Weighted  Percentages  and  Composite  Scale  Means 
Weighted  Means , Standard  Deviations , and  Percentiles 
Revised  Results 


MEAN 

STD 

DEV 

10TH 

25TH 

50TH 

75TH 

90TH 

Alabama. 

252 . 2 ( 

1.7) 

35. 5( 

1.2) 

206. 8( 

1.8) 

227. 9( 

1.9) 

251. 6{ 

1.7) 

276. 5{ 

1.7) 

299. 2( 

2.3) 

Arizona 

265. 4( 

1.3) 

32. 4( 

0.6) 

223. 3( 

2.3) 

243. 9( 

1.4) 

265. 9< 

1.8) 

287. 6 < 

1.1) 

306. 8( 

1.2) 

Arkansas 

256. 3( 

1.2) 

33. 9< 

0.6) 

212. 1( 

1.7) 

234. 1( 

1.4) 

257. 2 { 

1.2) 

279. 5 ( 

1.4) 

299. 0( 

1.2) 

California 

260.9 ( 

1.7) 

38. 3( 

1.1) 

210. 1{ 

1.8) 

234. 8( 

2.8) 

262. 3( 

1.4) 

288. 5( 

2.2) 

309. 0( 

2.2) 

Colorado 

272. 4( 

1.0) 

32. 7< 

0.6) 

228. 8( 

1.8) 

251. 1{ 

1.1) 

274. 1( 

1.2) 

295. 4( 

1.1) 

313. 5( 

1.1) 

Connecticut 

273.7 ( 

1.1) 

35. 6( 

0.9) 

225. 4t 

2.6) 

249. 8( 

1.4) 

276. 1{ 

0.9) 

299. 6( 

0.8) 

318. 4< 

1.3) 

Delaware 

262. 9( 

1.0) 

35. 3( 

0.7) 

217 .7 ( 

2.4) 

239. 7( 

1.2) 

263. 3( 

1.2) 

287. 2 ( 

1.7) 

307. 3( 

1.5) 

District  of  Columbia 

234. 9( 

0.9) 

36. 2( 

0.9) 

189. 9( 

1.1) 

210. 0( 

1.2) 

234.3 { 

1.9) 

258. 1{ 

2.1) 

280. 8< 

1.6) 

Florida 

259. 9< 

1.5) 

36. 4( 

0.8) 

211. 5( 

2.9) 

235. 1( 

1.7) 

260. 9( 

1.9) 

285. 5( 

1.5) 

306. 7( 

1.8) 

Georgia 

259. 4< 

1.2) 

34. 3{ 

0.6) 

215. 0( 

1.4) 

235. 8( 

1.2) 

260. 3( 

1.3) 

283. 6( 

2.1) 

303. 3( 

1.3) 

Hawaii 

257. 4< 

0.9) 

37. 3( 

0.6) 

209. 4( 

2.5) 

232. 0( 

0.9) 

258. 3( 

1.4) 

283. 4< 

1.0) 

305. 0( 

1.6) 

Idaho 

275. 1( 

0.7) 

30. 1( 

0.5) 

236. 1( 

1.1) 

255. 4< 

1.0) 

276. 0( 

1.0) 

296. 1{ 

0.9) 

313. 1( 

1.1) 

Indiana 

270. 1( 

1.1) 

33. 5( 

0.6) 

226.4 ( 

1.4) 

247. 8( 

1.5) 

270. 6{ 

1.5) 

293. 2( 

1.5) 

313. 4( 

3.2) 

Iowa 

283. 4< 

1.0) 

29. 5( 

0.6) 

245. 0< 

2.6) 

263. 4( 

1.2) 

284. 3( 

1.3) 

304. 2< 

1.8) 

320. 6{ 

1.3) 

Kentucky 

262 . 2 ( 

1.1) 

33. 9( 

0.6) 

217. 6 ( 

1.8) 

239. 5( 

1.6) 

262. 9( 

1.1) 

285. 2< 

1.4) 

305. 6( 

2.3) 

Louisiana 

250. 0( 

1.7) 

33. 7( 

0.8) 

205. 8( 

2.7) 

227. 4< 

1.9) 

250. 6< 

1.8) 

272.9 ( 

1.8) 

293. 5( 

1.5) 

Maine 

278. 6( 

1.0) 

30. 3( 

0.8) 

240. 3( 

2.0) 

259. 1{ 

1.1) 

279. 4( 

1.3) 

299. 4( 

1.5) 

316. 5( 

1.7) 

Maryland 

264. 8 ( 

1.3) 

38. 9( 

0.9) 

214*1 ( 

2.6) 

237. 9 ( 

2.1) 

265. 9( 

0.9) 

292 . 9 ( 

1.2) 

313. 6( 

1.9) 

Massachusetts 

272. 3( 

1.0) 

33. 8< 

0.7) 

229. 7( 

1.4) 

249. 6( 

2.2) 

273.7 ( 

1.3) 

297. 2( 

1.6) 

315. 7( 

1.5) 

Michigan 

267. 4( 

1.4) 

35. 0( 

0.6) 

221. 2{ 

1.1) 

244. 1< 

2.1) 

269.lt 

1.4) 

292. 3( 

2.3) 

311. 4( 

2.2) 

Minnesota 

282. 4{ 

1.0) 

31. 5( 

0.5) 

241. 1( 

1.7) 

261. 1( 

1.2) 

283. 6 ( 

1.4) 

304. 5< 

1.3) 

322. 2( 

1.6) 

Mississippi 

246. 5( 

1.2) 

34. 5( 

0.6) 

201. 8( 

1.1) 

222. 5< 

1.1) 

246. 4< 

1.3) 

270. 8( 

1.5) 

291. 4{ 

1.3) 

Missouri 

271. 1( 

1.2) 

32. 3( 

0.7) 

229. 0( 

2.3) 

249. 6( 

1.9) 

272. 6( 

1.3) 

293.3 ( 

1.7) 

311. 6( 

1.4) 

Nebraska 

277. 7( 

1.1) 

32. 0( 

0.5) 

235. 2( 

1.8) 

257. 3 ( 

1.2) 

279. 8( 

1.5) 

300. 1( 

1.3) 

317. 0( 

1.2) 

New  Hampshire 

278. 2( 

1.0) 

30. 0( 

0.7) 

239. 0( 

1.3) 

258. 6< 

0.7) 

278. 7{ 

0.8) 

298. 9( 

1.3) 

316. 0{ 

1.7) 

New  Jersey 

271. 9( 

1.6) 

35. 6( 

0.9) 

223. 4( 

2.0) 

248. 0( 

2.3) 

273. 6< 

1.8) 

297. 2( 

2.0) 

317. 0{ 

2.1) 

New  Mexico 

259. 6( 

0.9) 

31. 9( 

0.7) 

218. 4( 

2.7) 

238. 5( 

1.1) 

260. 1( 

0.9) 

281. 6< 

1.1) 

300. 6( 

1.0) 

New  York 

266. 4( 

2.1) 

39. 0( 

1.3) 

214. 6< 

3.3) 

241. 9( 

2.5) 

269. 2( 

1.9) 

293 . 3 ( 

1.5) 

314. 6{ 

2.0) 

North  Carolina 

258. 4( 

1.2) 

35. 1( 

0.8) 

213. 4( 

2.1) 

234. 6( 

1.8) 

259. 3{ 

0.9) 

282. 6( 

1.2) 

302. 9( 

2.0) 

North  Dakota 

283. 2( 

1.1) 

27. 9 ( 

0.6) 

246. 0( 

1.3) 

264. 6< 

1.3) 

284. 9( 

1.0) 

302. 3( 

1.5) 

317. 9< 

1.6) 

Ohio 

268. 1{ 

1.5) 

34. 2( 

0.9) 

223. 1< 

2.3) 

245. 4( 

1.8) 

269. 9( 

1.5) 

292. 2( 

1.0) 

310. 6{ 

1.6) 

Oklahoma 

268. 1( 

1.1) 

31. 9( 

0.6) 

227. 0( 

1.1) 

248. 1< 

1.3) 

269. 1( 

1.1) 

290. 7( 

1.5) 

308. 1( 

1.3) 

Pennsylvania 

271. 4( 

1.5) 

34. 1{ 

0.9) 

226. 2( 

2.5) 

248. 9( 

1.3) 

273. 0{ 

1.3) 

295. 4( 

1.2) 

314. 6< 

1.9) 

Rhode  Island 

265. 9t 

0.7) 

33.1< 

0.5) 

222. 0< 

C . 9) 

243. 8( 

1.2) 

267. 6 ( 

1.1) 

289.lt 

1.5) 

307.1( 

1.1) 

South  Carolina 

260. 9 ( 

1.0) 

34. 9 ( 

0.7) 

216. 4< 

1.6) 

236. 3( 

1.2) 

260. 0( 

1.3) 

285. 2< 

1.7) 

307. 0( 

1.7) 

258 . 8 * 

1.4) 

33.  S< 

0.7) 

215. 3( 

2.2) 

236. 0< 

1.5) 

259. 1( 

1.6) 

282. 8{ 

1.8) 

301. 9( 

1.7) 

Texas 

264. 6 < 

1.3) 

36. 8( 

0.7) 

217. 0( 

2.0) 

238. 6( 

1.3) 

264. 8( 

1.6) 

290. 0< 

1.9) 

312. 0( 

2.1) 

Utah 

274. 3( 

0.7) 

31. 6( 

0.8) 

232. 7( 

1.3) 

253. 8( 

1.3) 

275. 7( 

0.9) 

296. 6( 

1.2) 

313. 9t 

1.2) 

Virginia 

267. 9( 

1.2) 

35.0  ( 

0.6) 

222 .5  ( 

1.7) 

244. 0( 

1.7) 

268. 2( 

1.7) 

291. 9( 

1.6) 

313. 5( 

1.8) 

West  Virginia 

259. 1( 

1.0) 

30. 8< 

0.6) 

219. 0( 

1.8) 

238. 1{ 

1.1) 

259. 4< 

1.4) 

281. 1( 

0.9) 

298. 6( 

1.5) 

Wisconsin 

277. 9 ( 

1.5) 

32. 7{ 

0.7) 

234. 3( 

2.3) 

257. 6 { 

2.1) 

279. 8< 

1.3) 

300. 9( 

1.4) 

317 . 9 \ 

1.2) 

Wyoming 

275. 1( 

0.9) 

29. 2( 

0.5) 

237. 6( 

1.1) 

255. 1( 

1.1) 

276. 0( 

1.2) 

295. 4( 

1.0) 

312. 2( 

1.3) 

Guam 

235. 1( 

1.0) 

39. 2( 

0.9) 

184. 5( 

1.9) 

208. 1( 

3.3) 

234. 3 { 

1.9) 

262. 4< 

1.7) 

286. 7( 

1.8) 

Virgin  Islands 

222. B{ 

1.1) 

30. 0( 

0.6) 

184 . 2 ( 

1.7) 

202. 2< 

1.8) 

222 . 4 { 

1.5) 

243 . 0 ( 

1.6) 

260.9  ( 

2.0) 

BEST  COPY  AVAILABLE 


Table  G-7 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  4 Weighted  Percentages  and  Composite  Scale  Means 
Percent  of  Students  At  or  Above  the  Achievement  Levels 


N WXIOHTXD  PCT  [CVJ 


Alabama 

2605 

1.9  { 

0.1) 

3k] 

Arizona 

2741 

1.8  ( 

0.0) 

2k] 

Arkansas 

2621 

1.2  ( 

0.0) 

4k] 

California 

2412 

12.3  ( 

0.3) 

3k] 

Colorado 

2906 

1.7  ( 

0.1) 

3k] 

Connecticut 

2600 

1.2  ( 

0.0) 

3k] 

Delaware 

2040 

0.3  ( 

0.0) 

0\] 

District  of  Columbia 

2399 

0 . 2 ( 

0.0) 

0k] 

Florida 

2828 

5.4  ( 

0.2) 

3k] 

Gaorgia 

2766 

3 .4  < 

0.1) 

4k] 

Hawaii 

2625 

0 . 5 ( 

0.0) 

2k] 

Idaho 

2784 

0 . 6 ( 

0.0) 

3k) 

Indiana 

2593 

2 .7  ( 

0.1) 

3k] 

Iowa 

2770 

1 .3  ( 

0.0) 

4k] 

Xentucky 

2703 

1 • 8 ( 

0.1) 

4k] 

Louisiana 

2792 

2 . 2 ( 

0.1) 

3k] 

Maina 

1898 

0 . 6 ( 

0.0) 

3k] 

Maryland 

2844 

2 . 0 ( 

0.1) 

3k] 

Massachusetts 

2549 

2 . 1 { 

0.1) 

4k] 

Michigan 

2412 

4.1( 

0.1) 

4k] 

Hinnasota 

2640 

2 .1  ( 

0.1) 

4k] 

Hiaaiaaippi 

2712 

1.4  ( 

0.0) 

3k] 

Miaaouri 

2509 

2 . 1 ( 

0.1) 

5k) 

Nebraska 

2327 

0 .7  { 

0.0) 

3k] 

Naw  Hampahiro 

2265 

0 . 5 { 

0.0) 

3k] 

Haw  Jaraay 

2231 

2 . 8 ( 

0.1) 

4k] 

Naw  Haxico 

2342 

0 . 8 ( 

0.1) 

6k] 

Naw  York 

2284 

6 .7  ( 

0.2) 

3k] 

North  Carolina 

2884 

3 . 0 { 

0.1) 

3kl 

North  Dakota 

2193 

0 , 3 ( 

0.0) 

4k] 

Ohio 

2637 

4 .9  { 

0.1) 

3k] 

Oklahoma 

2254 

1.6  ( 

0.0) 

3k] 

Pennsylvania 

2740 

4 . 6 ( 

0.2) 

4k] 

Rhode  Island 

2390 

0 .4  { 

0.0) 

5k) 

South  Carolina 

2771 

1 . 8 ( 

0.1) 

3k) 

Tenmasaee 

2708 

2 .4  ( 

0.1) 

3k] 

Texas 

2623 

9 . 0 ( 

0.3) 

4k] 

Utah 

2799 

1.3  ( 

0.0) 

2k] 

Virginia 

2786 

2 . 8 ( 

0.1) 

3k] 

Neat  Virginia 

2786 

0 . 9 { 

0.0) 

4k  1 

Wisconsin 

2780 

2 « 1 ( 

0.1) 

4k] 

Wyoming 

2605 

0 . 3 ( 

0.0) 

3k] 

Guam 

1933 

0 . 1 ( 

0.0) 

0k] 

Virgin  Islands 

905 

0 . 1 ( 

0.0) 

0k] 

Original  Results 


KZAH 

ADVANCXD 

PRFCIiaiT 

BASIC 

206. 9( 

1.6) 

0 . 6 { 

0.2) 

10. 5 ( 

1.3) 

44. 7( 

2.2) 

213.8 ( 

1.1) 

1-2  ( 

0.3) 

13. 5( 

0.9) 

55. 2( 

1.7) 

208. 7( 

0.9) 

0.6  ( 

0.2) 

10. 0( 

0.8) 

48. 8( 

1.3) 

207. 1( 

1.6) 

1.6  ( 

0.5} 

12. 7{ 

1.2) 

48. 0( 

2.0) 

219. 8< 

l.C) 

2 . 1 ( 

0.4) 

18. 2{ 

1.1) 

62. 5 ( 

1.4) 

225 . 8 ( 

1.2) 

3 . 7 ( 

0.6) 

25. 2( 

1.4) 

68. 8< 

1.5) 

216. 6< 

0.8) 

2.3  ( 

0.4) 

17. 0( 

0.8) 

56. 2( 

1.0) 

191. 2{ 

0.5) 

1.0( 

0.2) 

5 .5  ( 

0.3) 

24 . 9 ( 

1.0) 

212 . 4 ( 

1.5) 

1 .5  ( 

0.4) 

13. 5( 

1.4) 

53. 4( 

2.0) 

214. 3( 

1.3) 

1.5  ( 

0.4) 

15. 7{ 

1.2)  - 

54. 9( 

1.7) 

212. 8( 

1.3) 

1.5< 

0.4) 

15. 2( 

1.0) 

53. 7 < 

1.8) 

220 . 3 ( 

1.0) 

1.1( 

0.3) 

16. 1( 

1.1) 

64. 2 ( 

1.7) 

219. 7( 

1.1) 

1.5  ( 

0.3) 

16. 2< 

1.1) 

61. 6< 

1.6) 

229. 0( 

1.1) 

3 .1  ( 

0.5) 

27. 0( 

1.3) 

74. 1< 

1.4) 

213. 6( 

1.0) 

1.4  { 

0.5) 

12.9  { 

1.1) 

52  -7  ( 

1.5) 

202. 8{ 

1.4) 

0 . 6 ( 

0.2) 

7 . 9 ( 

0.8) 

40. 5 ( 

2.0) 

230. 7( 

1.0) 

3 .0  ( 

0.6) 

28. 3< 

1.5) 

76. 3( 

1.3) 

216. 1( 

1.3) 

2.8< 

0.4) 

18.9  ( 

1.2) 

56. 8( 

1.6) 

225. 5( 

1.2) 

3.1< 

0.5) 

24.0  ( 

1.5) 

69. 7( 

1.6) 

218. 6 ( 

1.8) 

1 .7  C 

0.5) 

19. 1{ 

1.7) 

62. 1< 

2.2) 

227. 5{ 

0.9) 

3 . 4 ( 

0.5) 

26. 8( 

1.2) 

71. 8( 

1.4) 

200. 1( 

1.1) 

0 .4  { 

0.1) 

6 . 5 ( 

0.7) 

37. 3< 

1.3) 

221. 0{ 

1.2) 

1.9  ( 

0.3) 

19. 3( 

1.3) 

63. 6< 

1.6) 

224.2 ( 

1.3) 

2 .6  ( 

0.5) 

22. 5< 

1.7) 

68. 1( 

1.8) 

228. 6( 

1.2) 

3 . 0 ( 

0.6) 

25. 7{ 

1.7) 

73. 8( 

1.6) 

226. 1( 

1.5) 

3 .2  ( 

0.7) 

25. 3( 

1.6) 

69. 9( 

2.1) 

211. 8( 

1.5) 

1.0( 

0.4) 

11. 4{ 

1.3) 

51. 6< 

1.9) 

217. 2{ 

1.3) 

2 . 0 ( 

0.3) 

17  .4  ( 

1.3) 

58. 8{ 

1.9) 

211. 4( 

1.1) 

1 . 6 ( 

0.4) 

13. 2( 

0.9) 

52. 1( 

1.6) 

227. 6{ 

0.8) 

1.8  ( 

0.3) 

22. 9( 

1.1) 

74. 2( 

1.2) 

217. 5( 

1.2) 

1. 9 ( 

0.3) 

16. 8 { 

1.1} 

58. 8( 

1.7) 

219. 0{ 

1.0) 

1.2  ( 

0.4) 

14. 4( 

1.1) 

61. 5< 

1.6) 

223. 2< 

1.4) 

2 . 6 { 

0.5) 

22. 5( 

1.5) 

66. 4( 

1.9) 

214. 0< 

1.6) 

1. 6 ( 

0.4) 

13. 8( 

1.2) 

55. 8( 

2.2) 

211. 1( 

1.1) 

1. 2 ( 

0.3) 

13. 3{ 

1.1) 

49. 3( 

1.5) 

209 .4  { 

1.4) 

0.7  ( 

0.2) 

10. 2( 

1.0) 

49. 0< 

2.1) 

216. 6( 

1.3) 

1.8  ( 

0.5) 

15. 5< 

1.3) 

57. 6 { 

1.7) 

222. 8( 

1.0) 

1 . 9 ( 

0.3) 

19. 5 { 

1.1) 

67. 4 ( 

1.6) 

219. 6( 

1.3) 

3 . 1 ( 

0.7) 

19. 4( 

1.6) 

60. 3< 

1.4) 

213. 9( 

1.1) 

1.3  ( 

0.3) 

12. 8( 

1.0) 

54. 1< 

1.6) 

227. 7 ( 

1.1) 

3 . 0 ( 

0.5) 

25. 3( 

1.4) 

72. 4{ 

1.3) 

224. 2( 

1.0) 

1 * 5 ( 

0.3) 

19. 4< 

1.2) 

70. 1( 

1.4) 

191. 1( 

0.8) 

0 .4  { 

0.1) 

4 . 9 ( 

0.5) 

28. 0( 

1.2} 

178. 0( 

1.2) 

0.0( 

0.0) 

0 . 4 ( 

0.2) 

11. 8( 

1.6) 

< BASIC 


55. 3(  2.2) 
44. 8(  1.7} 
S1.2 ( 1.3) 
52. 0(  2.0) 
37. 5(  1.4) 
31. 2{  1.5) 

43. 8(  1.0) 
75. 1(  1.0) 
46. 6<  2.0) 
45. 1(  1.7) 
46. 3(  1.8) 
35. 8<  1.7) 

38. 4(  1.6) 
25. 9{  1.4) 
47. 3(  1.5) 
59. 5(  2.0) 
23. 7(  1.3) 
43. 2(  1.6) 

30. 3(  1.6) 
37. 9{  2.2) 
28. 2(  1.4) 
62.7  ( 1.3) 
36. 4<  1.6) 
31. 9{  1.8) 

26. 2<  1.6) 
30. 1(  2.1) 
48. 4(  1.9) 
41. 2{  1.9) 
47. 9{  1.6) 
25. 8(  1.2) 

41. 2(  1.7) 
38. 5(  1.6) 
33. 6(  1.9) 
44. 2(  2.2) 
50. 7(  1.5) 
51. 0{  2.1) 

42. 4(  1.7) 
32. 6{  1.6) 
39. 7<  1.4) 
45. 9(  1.6) 
27. 6 { 1.3) 
29. 9(  1.4) 

72. 0(  1.2) 

88. 2{  1.6) 


din 


Table  G-8 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  4 Weighted  Percentages  and  Composite  Scale  Means 
Percent  of  Students  At  or  Above  the  Achievement  Levels 
Revised  Results 


N 


Alabama 

2605 

Arizona 

2741 

Arkansas 

2621 

California 

2412 

Colorado 

2906 

Connecticut 

2600 

Delaware 

2040 

District  of  Columbia 

2399 

Florida 

2828 

Georgia 

2766 

Hawaii 

2625 

Idaho 

27  84 

Indiana 

2593 

Iowa 

2770 

Kentucky 

2703 

Louisiana 

2792 

Maine 

1898 

Maryland 

2844 

Massachusetts 

2549 

Michigan 

2412 

Minnesota 

2640 

Mississippi 

2712 

Missouri 

2509 

Nebraska 

2327 

New  Hampshire 

2265 

New  Jersey 

2231 

New  Mexico 

2342 

New  York 

2284 

North  Carolina 

2884 

North  Dakota 

2193 

Ohio 

2637 

Oklahoma 

2254 

Pennsylvania 

2740 

Rhode  Island 

2390 

South  Carolina 

2771 

2708 

Texas 

2623 

Utah 

2799 

Virginia 

2786 

West  Virginia 

2786 

Wisconsin 

2780 

Wyoming 

2605 

Guam 

1933 

Virgin  Islands 

905 

IGJHTXD 

PCT 

tcv] 

MEAN 

1 • 9 { i 

0.1) 

I 3X] 

208. 3( 

1.6) 

1 . 8 ( 

0.0) 

[ 2* ) 

215. 3{ 

1.1) 

1 . 2 { 

0.0) 

[ 4\) 

210. 2( 

0.9) 

12. 3( 

0.3) 

t 3XJ 

208. 4( 

1.6) 

1.7  ( 

0.1) 

[ 3X] 

221. 0( 

1.0) 

1 • 2 { 

0.0) 

t 3X) 

226. 8< 

1.1) 

0 . 3 ( 

0.0) 

[ 0M 

217.9 ( 

0.8) 

0 . 2 ( 

0.0) 

C OX] 

192. 6( 

0.5) 

5 . 4 < 

0.2) 

[ 3%) 

213. 7( 

1.5) 

3 .4  ( 

0.1) 

[ 4X] 

215. 6( 

1.2) 

0.5( 

0.0) 

C 2X] 

214. 1( 

1.3) 

0 . 6 ( 

0.0) 

l 3X] 

221. 6{ 

1.0) 

2 . 7 ( 

0.1) 

[ 3X] 

221. 0( 

1.0) 

1.3  ( 

0.0) 

t 4X] 

229. 9( 

1.0) 

1 . 8 ( 

0.1) 

I 4\] 

215. 0( 

1.0) 

2 . 2 ( 

0.1) 

C 3*] 

204. 1( 

1.5) 

0 . 6 { 

0.0) 

[ 3X] 

231. 6( 

1.0) 

2 « 0 < 

0.1) 

t 3X] 

217. 3( 

1.3) 

2 . 1( 

0.1) 

[ 4X] 

226. 6( 

1.2) 

4.1< 

0.1) 

t 4X] 

219. 9( 

1.7) 

2 .1  ( 

0.1) 

I 4X1 

228. 5( 

0.9) 

1.4  ( 

0.0) 

t 3X] 

201. 8( 

1.1) 

2 . 1( 

0.1) 

t 5%) 

222. 2< 

1.2) 

0-7  ( 

0.0) 

[ 3X] 

225. 3( 

1.2) 

0 . 5 ( 

0.0) 

[ 3X] 

229. 7( 

1.2) 

2 . 8 ( 

0.1) 

[ 4X] 

227. 1< 

1.5) 

0 . 8 ( 

0.1) 

t 6\] 

213. 3( 

1.4) 

6 .7  { 

0.2) 

[ 3X) 

218. 4{ 

1.2) 

3 . 0 ( 

0.1) 

t 3X) 

212. 9( 

1.1) 

0 . 3 < 

0.0) 

t 4X) 

228.7 ( 

0.8) 

4 . 9 ( 

0.1) 

[ 3X] 

218. 7( 

1.2) 

1 . 6 ( 

0.0) 

t 3X] 

220. 3( 

1.0) 

4 . 6 ( 

0.2) 

[ 4X1 

224. 3( 

1.3) 

0 . 4 ( 

0.0) 

t 5X) 

215. 4( 

1.5) 

1.8  ( 

0.1) 

t 3X] 

212 .51 

1.1) 

2 . 4 ( 

0.1) 

[ 3X] 

210. 9( 

1.4) 

9 . 0 ( 

0.3) 

[ 4X] 

217. 9 < 

1.2) 

1 .3  ( 

0.0) 

I 2X1 

224. 0{ 

1.0) 

2 .8  ( 

0.1) 

[ 3X] 

220.8  ( 

1.3) 

0 . 9 ( 

0.0) 

r 4X] 

215. 3( 

1.1) 

2 . 1 { 

0.1) 

1 4X] 

228.7  ( 

1.1) 

0 . 3 ( 

0.0) 

l 3X) 

225. 4( 

0.9) 

o.u 

; o.o) 

[ OX] 

192. 8( 

: 0.8) 

0.1< 

; o.o) 

t oxj 

17  8 . 9 ( 

: i.2) 

ADVANCED 

P RFC I ENT 

BASIC 

0 . 5 ( 0.1) 

10. 1(  1.2) 

43. 0<  2.1) 

0 . 8 ( 0.2) 

13. 1(  0.9) 

53. 5(  1.6) 

0 . 4 ( 0.2) 

9.7 ( 0.7) 

46. 9(  1.5) 

1. 3 ( 0.4) 

12. 4(  1.2) 

46. 4<  1.9) 

1 . 5 { 0.4) 

17. 5(  1.0) 

60. 8(  1.4) 

2 . 9 ( 0.5) 

24. 4(  1.4) 

67. 3(  1.6) 

1. 8 ( 0.3) 

16. 5(  0.9) 

54. 6{  1.0) 

0 . 9 ( 0.2) 

5 . 5 ( 0.3) 

23. 1(  0.9) 

1.2 ( 0.3) 

13. 3(  1.4) 

51. 6(  1.7) 

1.1 ( 0.3) 

15. 3(  1.2) 

53. It  1.7) 

1.1(  0.2) 

14.61  0.9) 

51. 9(  1.8) 

0 .7 ( 0.3) 

15. 7{  1.0) 

62. 7(  1.7) 

1.0(  0.2) 

15. 6(  1.1) 

59. 8(  1.7) 

2 • 3 ( 0.4) 

26. 0(  1.2) 

72. 4(  1.5) 

1.0<  0.3) 

12. 6(  1.2) 

50. 9(  1.5) 

0 .4 ( 0.2) 

7 . 6 ( 0.8) 

38. 8(  2.0) 

2 . 4 ( 0.5) 

27. 4<  1.5) 

74. 8(  1.5) 

2 . 2 ( 0.3) 

18. 4(  1.2) 

55. 1(  1.6) 

2 .4 ( 0.5) 

23. 3(  1.5) 

68. 5(  1.6) 

1 .2 ( 0.4) 

18. 5(  1.7) 

60. 5{  2.2) 

2 . 5 < 0.4) 

25. 9(  1.3) 

70. 6(  1.6) 

0 . 3 ( 0.1) 

6 . 3 ( 0.6) 

35. 8(  1.3) 

1.3 ( 0.3) 

18. 6(  1.3) 

62. 1(  1.7) 

2 . 0 ( 0.5) 

21. 8<  1.6) 

66. 6(  1.8) 

2.1<  0.4) 

24. 9(  1.6) 

72. 3(  1.6) 

2 . 5 ( 0.6) 

24. 6(  1.5) 

68. 2(  2.1) 

0 . 6 ( 0.2) 

11. 1<  1-3) 

49. 8(  2.0) 

1 . 5 { 0.3) 

17. 0(  1.3) 

57. 0(  1.8) 

1.2<  0.3) 

12. 7(  0.8) 

50. 3<  1.6) 

1 . 3 ( 0.3) 

22. 2(  1.1) 

72. 5<  1.3) 

1 .4 ( 0.3) 

16. 1(  1.2) 

57. 0{  1.7) 

0 . 8 ( 0.3) 

14. 0(  1.2) 

59. 5(  1.7) 

2 . 0 ( 0.4) 

21. 8{  1.5) 

64. 9 ( 2.0) 

1 . 2 ( 0.4) 

13. 3(  1.1) 

54. 2(  2.2) 

0 . 9 ( 0.3) 

12. 8(  1.1) 

47. 6(  1.7) 

0 . 5 ( 0.2) 

9 . 9 ( 1.0) 

47. 3(  2.0) 

1.2 ( 0.3) 

15. 0(  1.2) 

55. 7(  1.6) 

1 .4 ( 0.3) 

19. 0(  1.1) 

65. 6(  1.7) 

2 . 3 ( 0.5) 

10. 8(  1.5) 

58. 6(  1.4) 

1.0<  0.3) 

12. 2{  0.9) 

52. 4(  1.5) 

2 . 2 ( 0.4) 

24. 5(  1.4) 

70. 9(  1.4) 

1.0<  0.3) 

18. 7<  1.1) 

68. 6(  1.4) 

0 . 3 < 0.2) 

4 .7 ( 0.5) 

26. 3(  1.4) 

0.0(  0.0) 

0 . 5 ( 0.2) 

1C .6 ( 1.4) 

< BASIC 

57. 0<  2,1) 
46. 5(  1.6) 
53. 1(  1.5) 
53. 6(  1.9) 
39. 2(  1.4) 
32. 7(  1.6) 

45. 4(  1.0) 
76. 9(  0.9) 
48. 4<  1.7) 
46. 9(  1.7) 
48. 1(  1.8) 
37. 3(  1.7) 

40. 2<  1.7) 
27. 6(  1.5) 
49. 1(  1.5) 
61. 2<  2.0) 
25. 2(  1.5) 
44. 9(  1.6) 

31. 5<  1.6) 
39. 5(  2.2) 
29. 4(  1.6) 
64. 2(  1.3) 
37. 9(  1.7) 
33. 4<  1.8) 

27. 7<  1.6) 
31. 8<  2.1) 
50. 2(  2.0) 
43. 0(  1.8) 
49. 7(  1.6) 
27. 5<  1.3) 

43. 0(  1.7) 
40. 5(  1.7) 
35. 1(  2.0) 
45. 8(  2.2) 
52. 4<  1.7) 
52. 7<  2.0) 

44. 3(  1.6) 
34. 4(  1.7) 
41. 4(  1.4) 
47. 6(  1.5) 
29. 1(  1.4) 
31. 4(  1.4) 

73. 7(  1.4) 
89. 4<  1.4) 


BEST  COPY  AVAIL  ABU 


Table  G-9 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  8 Weighted  Percentages  and  Composite  Scale  Means 
Percent  of  Students  At  or  Above  the  Achievement  Levels 
Original  Results 


N 

WEIGHTED  PCT 

CCV] 

MEAN 

ADVANCED 

FRFCIENT 

BASIC 

< BASIC 

Alabama 

1992 

2522 

2 . 0 ( 

0.1) 

3k] 

251. 3( 

1.7) 

1.1( 

0.3) 

12. 3( 

1.1) 

44. 3( 

2.0) 

55. 7( 

2.0) 

1990 

2531 

2. 2( 

0.1) 

3k] 

252. 9{ 

1.1) 

1.  It 

0.2) 

11.7  ( 

0.8) 

47. 2( 

1.6) 

52. 8( 

1.6) 

Arizona 

1992 

2617 

1*7  ( 

0.1) 

4k] 

264. 6 ( 

1.3)  > 

1.8( 

0.4) 

18. 8< 

1.4) 

61. 4( 

1 . 8)  > 

38. 6( 

1 . 8 ) < 

1990 

2558 

1 - 8 ( 

0.1) 

3k) 

259. 6( 

1.3) 

1.5( 

0.4) 

16. 1{ 

1.1) 

54. 8( 

1.8) 

45. 2( 

1.8) 

Arkansas 

1992 

2556 

1 • 2 { 

0.0) 

2*] 

255. 4( 

1.2) 

l.Ot 

0.3) 

12. 9{ 

1.0) 

50. 2( 

1.7) 

49. 6t 

1.7) 

1990 

2669 

1.5< 

0.0) 

2k] 

256. 2 ( 

0.9) 

0 . 9 ( 

0.2) 

12. 4( 

1.0) 

51. 2< 

1.3) 

48. 8< 

1.3) 

California 

1992 

2516 

12. 5 ( 

0.3) 

3k) 

260.1( 

1.7) 

2.6  ( 

0.7) 

19. 7( 

1.4) 

55. 1{ 

2.0) 

44. 9( 

2.0) 

1990 

2424 

14. 2< 

0.4) 

3k] 

256.3 ( 

1.3) 

2 . 0 ( 

0.4) 

15.9  ( 

1.3) 

50.9  ( 

1.6) 

49. 1( 

1.6) 

Colorado 

1992 

2799 

1 . 6 { 

0.0) 

2k] 

271. 7t 

1.1)> 

2 . 5 ( 

0.5) 

26. 2( 

1.3)  > 

69. 2 ( 

1 • 3 ) > 

30. 8( 

1. 3)  < 

1990 

2675 

1.7  { 

0.0) 

2k] 

267. 4( 

0.9) 

2 .2  ( 

0.4) 

21. fit 

1.0) 

64. 3( 

1.1) 

35. 7( 

1.1) 

Connecticut 

1992 

2613 

1 . 2 ( 

0.0) 

3k] 

273.1( 

1.1)> 

4. It 

0.6) 

30.1( 

1.1)> 

68. 9( 

1.4) 

31. 1( 

1.4) 

1990 

2672 

1«  4 ( 

0.0) 

3k] 

269. 9( 

1.0) 

3.9  ( 

0.4) 

26. 2( 

1.1) 

65.9  ( 

1.3) 

34. 1( 

1.3) 

Delaware 

1992 

1934 

0.3  ( 

0.0) 

0k] 

262.lt 

1.0) 

2 .5  ( 

0.4) 

18. 5( 

1.1) 

57. 0t 

1.2) 

43. 0t 

1.2) 

1990 

2110 

0.3  ( 

0.0) 

0k] 

260. 7{ 

0.9) 

2 . 1 ( 

0.5) 

18. 6( 

0.9) 

54. 5{ 

1.3) 

45. 5< 

1.3) 

Diat  of  Columbia 

1992 

1816 

0 . 2 ( 

0.0) 

0k] 

233. 9( 

0 . 9 ) > 

0 .7  ( 

0.2) 

5 . 6 ( 

1.0) 

26. 0t 

1 .3)  > 

74. 0( 

1 . 3)  < 

1990 

2135 

0 . 2 { 

0.0) 

0k] 

231. 4( 

0.9) 

0 . 8 ( 

0.2) 

3 • 8 ( 

0.7) 

21.lt 

1.0) 

78. 9( 

1.0) 

Florida 

1992 

2549 

4 . 6 ( 

0.2) 

3k] 

259.1( 

1.5) 

1.8  ( 

0.4) 

17. 6 t 

1.3) 

54.7  ( 

1.9) 

45. 3( 

1.9) 

1990 

2534 

5.5( 

0.2) 

3k] 

255. 3( 

1.2) 

1.7  ( 

0.4) 

14. 8( 

1.0) 

49. 2( 

1.4) 

50. 8( 

1.4) 

Georgia 

1992 

2589 

3. 1( 

0.1) 

4k] 

258. 5( 

1.2) 

1.4  ( 

0.3) 

15.9  ( 

1.0) 

53. 5( 

1.5) 

46. 5< 

1.5) 

1990 

2766 

3 .7  < 

0.1) 

4k] 

258. 8( 

1.3) 

2 . 6 ( 

0.5) 

17. 2 ( 

1.3) 

53. 4( 

1.5) 

46. 6( 

1.5) 

Hawaii 

1992 

2454 

0 . 5 ( 

0.0) 

0k] 

256. 6( 

0 . 9 ) > 

2 .2  ( 

0.4) 

16. 5t 

0.8) 

51. 2{ 

1.2)  > 

48. 8( 

1. 2}  < 

1990 

2551 

0 . 5 ( 

0.0) 

0k] 

251. 0( 

0.8) 

1. 8 { 

0.3) 

14. 3( 

0.8) 

45. 3{ 

1.0) 

54. 7( 

1.0) 

Idaho 

1992 

2615 

0 . 7 ( 

0.0) 

2k] 

274. 4( 

0.8)> 

2 . 5 ( 

0.4) 

26. 9( 

1.2) 

73. 4< 

1.1) 

26. 6( 

1.1) 

1990 

2716 

0 . 8 { 

0.0) 

Ik] 

271. 4( 

0.8) 

1 • 5 ( 

0.4) 

23. 0( 

1.4) 

70.1( 

1.2) 

29. 9( 

1.2) 

Indiana 

1992 

2659 

3 . 0 { 

0.1) 

3k] 

269. 4( 

1.2) 

3 . 0 ( 

0.4) 

23. 8< 

1.3) 

65. 8( 

1.5) 

34. 2( 

1.5) 

1990 

2569 

3 . 2 ( 

0.1) 

3k) 

267. 3t 

1.2) 

2. 9 ( 

0.6) 

20. 6( 

1.2) 

63. 2( 

1.6) 

36.8  ( 

1.0) 

Iowa 

1992 

2816 

1 • 4 ( 

0.1) 

4k] 

282. 8( 

1.0)> 

4 . 7 ( 

0.7) 

37.2  ( 

1.4)  > 

81. 3( 

1 . 2 ) > 

18.7  ( 

1.2)  < 

1990 

2474 

1 . 5 ( 

0.1) 

4k] 

278. 0( 

1.1) 

3 . 8 ( 

0.5) 

30. 4< 

1.5) 

76. 3{ 

1.1) 

23. 7( 

1.1) 

Kentucky 

1992 

2756 

1. 8 { 

0.1) 

3k] 

261. 4 ( 

1.1)> 

1.9  ( 

0.4) 

16 .7  t 

1.1) 

57. 3( 

1 .3 ) > 

42. 7( 

1.3)  < 

1990 

2680 

2. 1( 

0.1) 

4k) 

257.1( 

1.2) 

1.2  ( 

0.2) 

13. 5< 

0.9) 

50. 5( 

1.8) 

49. 5( 

1.8) 

Louisiana 

1992 

2582 

1 * 9 ( 

0.1) 

4k] 

249.lt 

1.7) 

0 • 5 ( 

0.2) 

9 . 6 ( 

1.2) 

42. 4( 

2.0) 

57. 6 t 

2.0) 

1990 

2572 

2 . 2 ( 

0.1) 

4k] 

246. 4( 

1.2) 

0 . 6 ( 

0.2) 

7 . 6 ( 

1.0) 

38. 6t 

1.7) 

61. 4( 

1-7) 

Maine 

1992 

2464 

0 . 6 ( 

0.0) 

2k] 

278. 0t 

1.0) 

3 . 7 ( 

0.6) 

30. 8( 

1.9) 

77. 5( 

1.3) 

22. 5 ( 

1.3) 

1990 

0 

0.0( 

0.0) 

0k] 

( 

0.0) 

0.0) 

*****( 

0.0) 

*****( 

0.0) 

*****( 

0.0) 

Maryland 

1992 

2399 

1. 9 ( 

0.1) 

3k] 

264. 2( 

1.3) 

3 . 7 t 

0.6) 

23.8  ( 

1.3) 

58. 7( 

1.5) 

41. 3{ 

1.5) 

1990 

2794 

2 .2  ( 

0.0) 

2k] 

260. 8( 

1.4) 

3. It 

0.6} 

19.9  ( 

1.2) 

55. 8{ 

1.7) 

44. 2< 

1.7) 

Massachusetts 

1992 

2456 

2.1< 

0.1) 

4k] 

272.lt 

1.1) 

3 . 3 t 

0.5) 

27. 9 ( 

1.4) 

67.9  ( 

1.5) 

32. 1( 

1.5) 

1990 

0 

0 . 0 { 

0.0) 

0k) 

*•***( 

0.0) 

0.0) 

*****( 

0.0) 

**♦**< 

0.0) 

*****< 

0.0) 

Michigan 

1992 

2616 

4 . 2 ( 

0.1) 

2k] 

266. 6< 

1.4) 

2 . 6 ( 

0.5) 

23.1( 

1.7) 

63. 2< 

1.6) 

36. 8( 

1.6) 

1990 

2587 

4 . 8 ( 

0.1) 

3k] 

264. 4( 

1.2) 

2 . 4 ( 

0.4) 

19. 7( 

1.4) 

60. 3( 

1.4) 

39. 7( 

1.4) 

Minnesota 

1992 

2471 

2 - 0 { 

0.1) 

2k) 

281. 8< 

1.0)> 

5 . 6 1 

0 . 7 ) > 

36. 7( 

1 . 2)  > 

78. 5( 

1.2)  > 

21. 5( 

1 .2)  < 

1990 

2584 

2 .4  ( 

0.1) 

4k] 

275. 4( 

0.9) 

3 . 7 ( 

0.4) 

28. 7( 

1.2) 

73. 9< 

1.3) 

26.1( 

1.3) 

Mississippi 

1992 

2498 

1 • 4 ( 

0.0) 

3k] 

245. 5( 

1.2) 

0 . 5 ( 

0.2) 

8 . 5 1 

0.8) 

38. 3< 

1.5) 

61.7  ( 

1.5) 

1990 

0 

0.0( 

0.0) 

0k) 

***•*< 

0.0) 

0.0) 



0.0) 



0.0) 

***♦*( 

0.0) 

Missouri 

1992 

2666 

2 . 2 ( 

0.0) 

2k] 

270. 4( 

1.2) 

2 .7  ( 

0.4) 

23. 7( 

1.3) 

67. 9 t 

1.6) 

32.lt 

1.6) 

1990 

0 

0.0( 

0.0) 

0k] 

***** ( 

0.0) 

0.0) 

**»**! 

0.0) 

*****  ( 

0.0) 

♦****( 

0.0) 

> INDICATES  A SIGNIFICANT  INCREASE  (OR  DECREASE  "<■)  BETWEEN  1990  AND  1992 


Table  G-9  (continued) 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  8 Weighted  Percentages  and  Composite  Scale  Means 
Percent  of  Students  At  or  Above  the  Achievement  Levels 
Original  Results 


N WEIGHTED  PCT 

[CV] 

KEAN 

ADVANCED 

PRFCIENT 

BASIC 

< BASIC 

Nebraska 

1992 

1990 

2285 

2519 

0 . 8 ( 0.0) 
0 . 9 ( 0.0) 

t 3*) 
C 2k] 

277. 0{  1.1) 
275. 7(  1.0) 

3 . 6(  0.5) 
3 .6 ( 0.6) 

31. 7(  1.9) 
29. 9(  1.4) 

75. 4(  1.2) 
74. 3<  1.1) 

24. 6(  1.2) 
25. 7(  1.1) 

Nmr  Hampshira 

1992 

1990 

2535 

2568 

0 . 5 ( 0.0) 
0 . 5 { 0.0) 

I 2k] 
[ lk] 

277. 6(  1.0)> 
273. 1(  0.9) 

3 . 5 ( 0.6) 
3 .2 ( 0.5) 

30. 0(  1.5) > 
25. 0(  1.2) 

76. 8(  1.0)> 
71. 3(  1.6) 

23. 2(  1.0)< 
28. 7(  1.6) 

Naw  Jarsay 

1992 

1990 

2174 

2710 

3 . 2 ( 0.1) 
3 . 5 ( 0.1) 

[ 4\) 

t 4k] 

271. 2(  1.6) 
269. 7{  1.1) 

3 . 9 ( 0.6) 
3 . 9 ( 0.5) 

27. 6(  1.4) 
25. 3(  1.3) 

67. 3<  1.8) 
64. 5(  1.6) 

32. 7{  1.8) 
35. 5(  1.6) 

Haw  Maxi co 

1992 

1990 

2561 

2643 

0 . 8 < 0.0) 
0 . 8 ( 0.0) 

[ lk] 
t lk] 

258. 8<  0 . 9 ? > 
256. 4(  0.7) 

1 . 1 ( 0.3) 
1 . 2 ( 0.3) 

13. 8 ( 1.0) 
12. 8 { 0.9) 

54. 1{  1.4) 
50. 7{  1.3) 

45. 9<  1.4) 
49. 3(  1.3) 

Haw  York 

1992 

1990 

2158 

2302 

6 . 5 ( 0.2) 
7 . 7 { 0.2) 

t 3k] 
t 2k) 

265. 7(  2.1) 
260. 8(  1.4) 

3 .7 ( 0.6) 
3 . 1 ( 0.5) 

23. 8(  1 . 6 ) > 
18. 8(  1.0) 

62. 5(  2.3) 
56. 5(  1.7) 

37. 5(  2.3) 
43. 5(  1.7) 

North  Carolina 

1992 

1990 

2769 

2843 

3 . 2 ( 0.1) 
3 . 6 ( 0.1) 

[ 3k] 
[ 2k] 

257. 6(  1 . 2)  > 
250. 3(  1.1) 

1.5 ( 0.3) 
0 . 8 ( 0.4) 

15. 2(  1.0)> 
11. 3{  0.8) 

52. 8(  1 . 5) > 
44. 4(  1.4) 

47. 2<  1 . 5) < 
55. 6(  1.4) 

North  Dakota 

1992 

1990 

2314 

2485 

0 . 3 ( 0.0) 
0 . 4 ( 0.0) 

[ 3k] 
[ 4k] 

282. 6(  1.2) 
281. 1(  1.2) 

3 . 8 ( 0.6) 
3 . 9 < 0.6) 

35. 7(  1.7) 
33. 8(  2.0) 

82. 4(  1.3) 
81. 0(  1.6) 

17. 6(  1.3) 
19. 0<  1.6) 

Ohio 

1992 

1990 

2535 

2673 

5 . 4 { 0.2) 
5 . 6 { 0.1) 

[ 4k] 
C 2k] 

267. 4(  1.5) 
264. 0<  1.0) 

2 .4 ( 0.5) 
2 . 0 ( 0.3) 

22. 4{  1.4) 
18. 5(  1.2) 

64. 3{  2.0) 
59. 6(  1.4) 

35. 7(  2.0) 
40. 4(  1.4) 

Oklahoma 

1992 

1990 

2141 

22221' 

1.5 ( 0.0) 
1 . 8 ( 0.0) 

[ 3k] 
[ 2k] 

267. 4(  1 .2 ) > 
263. 2(  1.3) 

1.5 ( 0.3) 
1 . 6 ( 0.5) 

21. 1(  1 . 2 ) > 
16. 9(  1.3) 

65. 2(  2.0) 
59. 2(  1.6) 

34. 8(  2.0) 
40. 8(  1.6) 

Pennsylvania 

1992 

1990 

2612 

2528 

4 . 5 ( 0.1) 
5 .4 ( 0.2) 

[ 3k] 
[ 3k) 

270. 7(  1.5) 
266. 4{  1.6) 

3 . 2 ( 0.7) 
2 . 3 ( 0.4) 

25. 9(  1.5) 
21. 4(  1.5) 

67. 2(  1.7) 
62. 9(  2.0) 

32. 8(  1.7) 
37. 1{  2.0) 

Rhode  Island 

1992 

1990 

2120 

2675 

0 .4  { 0.0) 
0 . 4 ( 0.0) 

[ 0k] 
[ 2k] 

265. 1(  0 ./)> 
260. 0(  0.6) 

1 . 8 ( 0.3) 
1 . 8 ( 0.3) 

19. 7(  1.3) 
18. 3<  1.0) 

62. 0(  1.2) > 
54. 9(  0.9) 

38. 0(  1 .2) < 
45. 1(  0.9) 

South  Carolina 

1992 

1990 

2625 

0 

1 . 8 ( 0.0) 
0.0{  0.0) 

[ 2k] 
[ 0k] 

260. 0(  1.0) 
**•**(  o.O) 

2 . 1 ( 0.5) 

18. 0(  1.1) 

53. 4<  1.2) 

46. 6(  1.2) 

1992 

2485 

2 . 3 ( 0.1) 

[ 3k] 

258. 0(  1.4) 

1 . 2 ( 0.4) 

14. 7(  1.2) 

52. 6(  1.8) 

47. 4<  1.8) 

1990 

0 

0.0(  0.0) 

C 0k] 

**•**{  0.0) 

Texas 

1992 

1990 

2614 

2542 

8 . 8 ( 0.3) 
10. 2{  0.3) 

t 3k] 
[ 3k) 

263. 8(  1.3)  > 
258. 2(  1.4) 

3 . 6 ( 0.6) 
2 . 0 ( 0.4) 

21. 2(  1 . 4 ) > 
15. 9(  1.0) 

57. 8(  1 .5) > 
52. 0(  1.7) 

42. 2(  1 .5) < 
48. Q<  1.7) 

Utah 

1992 

1990 

2726 

0 

1 . 2 ( 0.0) 
0.0(  0.0) 

t 3k] 
l Ok) 

273. 6(  0.7) 
•****(  0.0) 

2 . 7 ( 0.5) 

27. 2(  1.1) 
*****(  0.0) 

72. 1(  1.3) 
**•**(  0.0) 

27. 9(  1.3) 

***•*(  o.O) 

Virginia 

1992 

1990 

2710 

2661 

2 . 8 ( 0.1) 
3 . 0 ( 0.1) 

t 4k] 

[ 3k] 

267. 1(  1.2) 
264. 3(  1.5) 

3 . 2 ( 0.5) 
4.1<  0.8) 

22. 9 ( 1.2) 
20. 7(  1.6) 

62. 3(  1.6) 
57. 7(  1.6) 

37. 7(  1.6) 
42. 3(  1.6) 

Hast  Virginia 

1992 

1990 

2690 

2600 

0 . 9 { 0.0) 
1.1{  0.0) 

t 3k] 
I 3k] 

258. 2(  1.0) 
255. 9{  1.0) 

0 . 7 ( 0.2) 
1.1 ( 0.2) 

12. 6(  0.9) 
12. 1(  0.9) 

52. 9(  1.5) 
49. 3(  1.2) 

47. 1<  1.5) 
50. 7(  1.2) 

Wisconsin 

1992 

1990 

2814 

2750 

2 . 3 ( 0.1) 
2 . 4 ( 0.1) 

l 6k] 
( 3k] 

277. 3(  1.5) 
274.5 { 1.3) 

3 . 9 ( 0.6) 
3 . 6 ( 0.5) 

32. 1(  1.4) 
28.ee  1.5) 

75. 6(  1.9) 
72. 3(  1.7) 

24. 4<  1.9) 
27. 7(  1.7) 

Wyoming 

1992 

1990 

2444 

2701 

0 . 3 { 0.0) 
0 . 3 ( 0.0) 

I 2k] 
[ lk] 

274. 4(  0 , 9 ) > 
272.1'  0.7) 

2 .4 ( 0.5) 
2 . 0 ( 0.3) 

25. 8(  1.0) 
23. 6{  1.0) 

73. 0(  1.3) 
71. 1(  1.3) 

27. 0(  1.3) 
28. 9<  1.3) 

Guam 

1992 

1990 

1496 

1617 

0 . 1 ( 0.0) 
0.1<  0.0) 

[ lk] 
l 0k] 

234. 3(  1.0)> 
231. 8(  0.7) 

0 . 6 ( 0.2) 
0 . 5 ( 0.2) 

7 . 0 ( 0.7) 
5 . 2 ( 0.6) 

29.5C  1.4) 
26. 7(  1.0) 

70. 5(  1.4) 
73. 3{  1.0) 

Virgin  Islands 

1992 

1990 

1479 

1326 

0 . 1 ( 0.0) 
0.1(  0.0) 

■ ( 0k] 
i t 0k] 

221. 9(  1.1)> 
218. 7(  0.9) 

0.0(  0.1) 
0 . 1 ( 0.1) 

0 . 9 ( 0.3) 
0 . 9 ( 0.4) 

12. 7(  1.0) 
10. 4(  1.1) 

87.31  1.0) 
89. 6(  1.1) 

> INDICATES  A 

SIGNIFICANT 

INCREASE 

(OR  DECREASE  ■<•) 

BETWEEN  1990  AND 

1992 

BEST  COPY  AVAILABLE 
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Alabama. 

Arizona 

Arkansas 

California 

Colorado 

Connecticut 

Delaware 

Dist  of  Columbia 

Florida 

Georgia 

Hawaii 

Idaho 

Indiana 

Iowa 

Kentucky 

Louisiana 

Maine 

Maryland 

Massachusetts 

Michigan 

Minnesota 

Mississippi 

Missouri 


Table  G-10 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  8 Weighted  Percentages  and  Composite  Scale  Means 
Percent  of  Students  At  or  Above  the  Achievement  Levels 
Revised  Results 


N WEIGHTED  PCT  [CV] 


MEAN 


ADVANCED  PRFCIKNT 


BASIC 


< BASIC 


1992 

2522 

2.0( 

0.1) 

[ 

3k] 

252. 2( 

1.7) 

0 . 9 ( 

0.3) 

10. 2( 

0.9) 

38. 8< 

1.9) 

61. 2( 

1.9) 

1990 

2531 

2.2< 

0.1) 

t 

3k] 

252. 9( 

1.1) 

1.0< 

0.2) 

9. 1< 

0.7) 

40. 3{ 

1.7) 

59. 7( 

1.7) 

1992 

2617 

1.7( 

0.1) 

t 

4k] 

265. 4( 

1.3)  > 

1.4  ( 

0.3) 

15. 1( 

1.3) 

54. 7{ 

1.8)> 

45. 3( 

1 . 8)  < 

1990 

2558 

1 • 8 ( 

0.1) 

i 

3k] 

259. 6( 

1.3) 

1 .3  ( 

0.4) 

12. 7( 

0.9) 

47. 5( 

1.8) 

52. 5( 

1.8) 

1992 

2556 

1.2< 

0.0) 

t 

2k] 

256. 3< 

1.2) 

0 . 7 ( 

0.2) 

10. 0( 

0.8) 

44. 3( 

1.8) 

55. 7( 

1.8) 

1990 

2669 

1.5( 

0.0) 

[ 

2k] 

256. 2( 

0.9) 

0 .7  ( 

0.2) 

9 .3  ( 

0.7) 

43.9  ( 

1.2) 

56. 1( 

1.2) 

1992 

2516 

12. 5 ( 

0.3) 

[ 

3k] 

260. 9< 

1.7)  > 

2.2  ( 

0.7) 

16. 2( 

1.3) 

50. 4( 

1.9) 

49. 6( 

1.9) 

1990 

2424 

14. 2( 

0.4) 

c 

3*] 

256. 3( 

1.3) 

1.7  { 

0.3) 

12. 5( 

1.1) 

44.6  ( 

1.7) 

55. 4( 

1.7) 

1992 

2799 

1.6( 

0.0) 

t 

2k] 

272. 4( 

1.0)> 

2 . 1 ( 

0.4) 

21. 6( 

1.2)  > 

63. 9( 

1.4)  > 

36. 1( 

1 .4 ) < 

1990 

2675 

1 *7  ( 

0.0) 

[ 

2k] 

267. 4{ 

0.9) 

1 • 9 ( 

0.4) 

16. 9( 

1.0) 

57. 5( 

1.2) 

42.5  ( 

1.2) 

1992 

2613 

1 * 2 ( 

0.0) 

t 

3*] 

273. 7( 

1.1)> 

3 .2  ( 

0.6) 

25. 7( 

1.1)> 

64. «( 

1.4) 

35. 6( 

1.4) 

1990 

2672 

1 * 4 ( 

0.0) 

[ 

3k] 

269. 9( 

1.0) 

3 .4  ( 

0.4) 

21. 7( 

0.9) 

59. 9( 

1.4) 

40. 1( 

1.4) 

1992 

1934 

0 . 3 ( 

0.0) 

[ 

Ok] 

262. 9( 

1.0) 

2 -3  < 

0.4) 

15. 1( 

1.0) 

51. 6( 

1.2) 

48. 4( 

1.2) 

1990 

2110 

0.3( 

0.0) 

[ 

0k] 

260. 7( 

0.9) 

1*9  ( 

0.4) 

14. 2( 

0.8) 

47. 8( 

1.5) 

52. 2( 

1.5) 

1992 

1816 

0 . 2 ( 

0.0) 

t 

0k] 

234.9  ( 

0.9)  > 

0 . 6 ( 

0.2) 

4 .4  ( 

0.9) 

21. 8( 

1.1)> 

78. 2< 

1.1X 

1990 

2135 

0 . 2 ( 

0.0) 

t 

0k] 

231. 4( 

0.9) 

0 . 8 ( 

0.2) 

3 .1  ( 

0.6) 

16. 6( 

1.0) 

83. 4{ 

1.0) 

1992 

2549 

4 . 6 ( 

0.2) 

t 

3k] 

259. 9( 

1 . 5 ) > 

1 . 5 ( 

0.3) 

14. 6( 

1.2) 

48. 8( 

1.9)> 

51. 2( 

1 . 9 ) < 

1990 

2534 

5.5< 

0.2) 

c 

3k] 

255. 3( 

1.2) 

1 . 4 { 

0.3) 

12. 0( 

0.9) 

42. 7< 

1.4) 

57. 3 ( 

1.4) 

1992 

2589 

3 . 1 ( 

0.1) 

[ 

4k] 

259. 4( 

1.2) 

1 . 1 ( 

0 . 3 ) < 

12. 7( 

0.9) 

47.9  ( 

1.7) 

52. 1( 

1.7) 

1990 

2766 

3 *7  ( 

0.1) 

c 

4k] 

258. 8( 

1.3) 

2 .4  ( 

0.4) 

13. 8( 

1.2) 

47.2  ( 

1.5) 

52. 8< 

1.5) 

1992 

2454 

0.5{ 

0.0) 

t 

Ok] 

257. 4( 

0 . 9 ) > 

1 . 8 { 

0.3) 

13. 5( 

0.7) 

46. 2( 

1.1)> 

53. 8( 

1.1>< 

1990 

2551 

0 . 5 { 

0.0) 

t 

0k] 

251. 0( 

0.8) 

1 .6  ( 

0.3) 

11. 7( 

0.7) 

40. 0( 

1.0) 

60. 0( 

1.0) 

1992 

2615 

0 . 7 ( 

0.0) 

t 

2k] 

275. 1( 

0.7)  > 

2 . 1 ( 

0.3) 

21.7  ( 

1.2) 

67.9  ( 

1.0)» 

32. 1( 

1.0X 

1990 

2716 

0 . 8 ( 

0.0) 

[ 

lk] 

271. 4{ 

0.8) 

1 . 2 ( 

0.3) 

17.9  ( 

1.1) 

63. 4( 

1.2) 

36. 6( 

1.2) 

1992 

2659 

3 . 0 ( 

0.1) 

[ 

3k] 

270. 1( 

1.1) 

2 . 6 ( 

0.4) 

19. 7( 

1.2) 

59. 9( 

1.5) 

40. 1( 

1.5) 

1990 

2569 

3.2( 

0.1) 

I 

3k] 

267. 3( 

1.2) 

2 . 5 ( 

0.5) 

16. 6< 

1.1) 

56. 4< 

1.5) 

43. 6( 

1.5) 

1992 

2816 

1.4< 

0.1) 

[ 

4k] 

283. 4( 

1.0)> 

3 . 9 { 

0.7) 

31. 2( 

1.3)  > 

76. 4< 

1.3)> 

23. 6( 

1.3)  < 

1990 

2474 

1.5< 

0.1) 

c 

4k] 

278. 0( 

1.1) 

3 . 3 ( 

0.5) 

25. 1( 

1.4) 

70. 0( 

1.2) 

30. 0( 

1.2) 

1992 

2756 

1 • 8 ( 

0.1) 

t 

3k] 

262. 2( 

1.1)> 

1.6< 

0.3) 

13. 8 ( 

1.1)> 

51. 2< 

1.5)> 

48. 8< 

1.5)  < 

1990 

2680 

2 . 1 ( 

0.1) 

l 

4k] 

257. 1( 

1.2) 

1.1( 

0.3) 

10. 5 ( 

0.8) 

43. 0< 

1.7) 

57.0  ( 

1.7) 

1992 

2582 

1.9< 

0.1) 

[ 

4k] 

250. 0{ 

1.7) 

0 . 5 ( 

0.2) 

7 . 2 ( 

1.0) 

36. 6( 

1.9) 

63. 4( 

1.9) 

1990 

2572 

2 . 2 ( 

0.1) 

[ 

4k] 

246. 4( 

1.2) 

0 . 5 ( 

0.2) 

5.4( 

0.6) 

31.7  ( 

1.6) 

68. 3< 

1.6) 

1992 

2464 

0 . 6 ( 

0.0) 

I 

2k] 

278. 6( 

1.0) 

3 . 1 ( 

0.6) 

25. 5( 

1.5) 

71. 6( 

1.3) 

28.4  ( 

1.3) 

1990 

0 

0.0( 

0.0) 

[ 

0k] 

*****( 

0.0) 

***•*( 

0.0) 

***•*< 

0.0) 

«****( 

0.0) 

*****{ 

0.0) 

1992 

2399 

1 . 9 ( 

0.1) 

( 

3k] 

264. 8( 

1. 3)  > 

3 . 2 ( 

0.5) 

19. 9( 

1.2) 

53. 9( 

1.4) 

46. 1( 

1.4) 

1990 

2794 

2 . 2 ( 

0.0) 

[ 

2k] 

260. 8( 

1.4) 

2 . 6 ( 

0.5) 

16. 6( 

1.2) 

49. 7< 

1.6) 

50. 3( 

1.6) 

1992 

2456 

2.1( 

0.1) 

t 

4k] 

272. 8( 

1.0) 

2 . 8 ( 

0.5) 

23. 3( 

1.3) 

62. 8( 

1.5) 

37. 2( 

1.5) 

1990 

0 

0.0( 

0.0) 

t 

0k] 

***•*( 

0.0) 

*****< 

0.0) 

*****( 

0.0) 

*****( 

0.0) 

***•*( 

0.0) 

1992 

2616 

4 . 2 ( 

0.1) 

t 

2k] 

267. 4( 

1.4) 

2 . 2 ( 

0.4) 

18. 9( 

1.5) 

57.9  ( 

1.7) 

42. 1( 

1.7) 

1990 

2587 

4 . 8 { 

0.1) 

[ 

3k] 

264. 4< 

1.2) 

2.1( 

0.4) 

15. 8( 

1.2) 

53. 3( 

1.7) 

46. 7( 

1.7) 

1992 

2471 

2 . 0 ( 

0.1) 

t 

2k] 

282. 4( 

1.0)> 

4 . 7 ( 

0.6) 

31. 1( 

1. 2)  > 

74. 2< 

1 .3)  > 

25. 8( 

1.3X 

1990 

2584 

2 . 4 ( 

0.1) 

t 

4k] 

275. 4( 

0.9) 

3 . 3 ( 

0.5) 

23. 3( 

1.2) 

67. 5< 

1.1) 

32. 5( 

1.1) 

1992 

2498 

1.4  ( 

0.0) 

[ 

3k] 

246. 5( 

1.2) 

0 . 3 { 

0.1) 

6.4( 

0.7) 

33. 4( 

1.6) 

€6 . 6 ( 

1.6) 

1990 

0 

0.0{ 

0.0) 

t 

0k] 

•****( 

0.0) 

**•**( 

0.0) 

*****( 

0.0) 

•****( 

0.0) 

*♦***( 

0.0) 

1992 

2666 

2 . 2 ( 

0.0) 

c 

2k] 

271 . 1 ( 

1.2) 

2 .3  ( 

0.4) 

19. 5< 

1.2) 

62. 5( 

1.6) 

37. 5( 

1.6) 

1990 

0 

0.0( 

0.0) 

t 

0k] 



0.0) 

0.0) 

*****  ( 

0.0) 

***•*( 

0.0) 

****♦( 

0.0) 

> INDICATES  A SIGNIFICANT  INCREASE  (OR  DECREASE  •<•)  BETWEEN  1990  AND  1992 


Table  G-10  (continued) 

NAEP  1992  Trial  State  Assessment  in  Mathematics 
Grade  8 Weighted  Percentages  and  Composite  Scale  Means 
Percent  of  Students  At  or  Above  the  Achievement  Levels 
Revised  Results 


N WEIGHTED  POT  [CVJ 

MEAN 

ADVANCED 

PRTCIKNT 

BASIC 

^ BASIC 

Nebraska 

1992 

2285 

0 . 8 ( 0.0)  [ 

3k] 

277. 7(  1.1) 

2 . 9 ( 0.5) 

26. 3(  1.6) 

70. 2<  1.3) 

29. 8(  1.3) 

1990 

2519 

0 . 9 ( 0.0)  [ 

2\) 

275. 7(  1.0) 

3 . 3 ( 0.5) 

24. 4<  1.2) 

68. 2(  1.3) 

31. 8(  1.3) 

New  Hampshire 

1992 

2535 

0 . 5 ( 0.0)  [ 

2k] 

278. 2(  1.0)> 

2 . 8 ( 0.5) 

24. 9(  1 . 4) > 

71. 2(  1 . 3 ) > 

28. 8(  1 . 3 ) < 

1990 

2568 

0.5{  0.0)  [ 

Ik] 

273. 1(  0.9) 

2 . 7 ( 0.5) 

20. 3(  1.2) 

64.7 ( 1.5) 

35. 3(  1.5) 

New  Jersey 

1992 

2174 

3 . 2 ( 0.1)  [ 

4k] 

271. 9<  1.6) 

3 . 2 ( 0.4) 

23. 7(  1.3) 

62. 5(  1.9) 

37. 5<  1.9) 

1990 

2710 

3 . 5 ( 0.1)  [ 

4k] 

269. 7(  1.1) 

3 . 4 ( 0.5) 

21. 3{  1.1) 

58. 2(  1.5) 

41. 8(  1.5) 

New  Mexico 

1992 

2561 

0 . 8 ( 0.0)  [ 

lk] 

259. 6t  0 . 9 ) > 

0 . 9 ( 0.3) 

10. 9(  0.8) 

47. 6(  1 . 3 ) > 

52. 4(  1 . 3 ) < 

1990 

2643 

0 . 8 < 0.0)  t 

IV 

256. 4(  0.7) 

1.0(  0.3) 

10. 2<  0.9) 

43. 2{  1.2) 

56. 8(  1.2) 

New  York 

1992 

2158 

6.5 { 0.2)  [ 

3k] 

266. 4(  2 . 1) > 

3 . 2 ( 0.5) 

20. 0(  1 . 3 ) > 

57. S(  2 . 2 ) > 

42. 5(  2 . 2 ) < 

1990 

2302 

7.7 ( 0.2)  [ 

2k] 

260. 8(  1.4) 

2 . 7 { 0.4) 

15.31  0.9) 

49. 9(  1.7) 

50. 1(  1.7) 

North  Carolina 

1992 

2769 

3 . 2 ( 0.1)  t 

3k] 

253. 4(  1.2) > 

1.2<  0.3) 

12.1 ( 1.0) > 

47. 0(  1.4) > 

53. 0(  1 . 4 ) < 

1990 

2843 

3 . 6 ( 0.1)  t 

2k] 

250. 3(  1.1) 

0 . 6 ( 0.3) 

8 . 7 ( 0.7) 

37. 9(  1.4) 

62. 1(  1.4) 

North  Dakota 

1992 

2314 

0 . 3 ( 0.0)  [ 

3k] 

283. 2(  1.1) 

3 . 1 ( 0.5) 

29. 5(  1.6) 

77. 9(  1.4) 

22.  If  1.4) 

1990 

2485 

0 . 4 < 0.0)  [ 

4k] 

281. 1(  1.2) 

3 . 6 { 0.6) 

27. 3{  1.8) 

75. 4<  1.6) 

24.  S{  1.6) 

Ohio 

1992 

2535 

5 . 4 ( 0.2)  L 

4k] 

268. 1<  1 .5) > 

1 . 9 ( 0.4) 

18. 1(  1.3) 

58. 8{  2.1) 

41. 2<  2.1) 

1990 

2673 

5 . 6 { 0.1)  t 

2k] 

264. 0<  1.0) 

1 . 7 ( 0.3) 

14.8 ( 1.1) 

52. 6<  1.6) 

47. 41  1.6) 

Oklahoma 

1992 

2141 

1 . 5 ( 0.0)  l 

: 3k] 

268. 1<  1.1)> 

l.K  0.3) 

16. 9{  1.1) 

59. 4(  1 . 6 ) > 

40. 6(  1.6)« 

1990 

2222 

1 . 8 ( 0.0)  I 

: 2k] 

263. 2(  1.3) 

1 .3 ( 0.4) 

13. 3(  1.2) 

52. 0<  1.8) 

48. 0(  1.8) 

Pennsylvania 

1992 

2612 

4 . 5 { 0.1)  1 

: 3k] 

271. 4(  1 . 5 ) > 

2 .7 ( 0.5) 

21. 5(  15) 

62. 1{  1.7) 

37. 9(  1.7) 

1990 

2528 

5 .4 ( 0.2)  | 

[ 3k] 

266. 4 ( 1.6) 

2 . 0 ( 0.4) 

17. 2 ( 1.3) 

56. 0{  2.0) 

44. 0(  2.0) 

Rhode  Island 

1992 

2120 

0 . 4 ( 0.0)  | 

[ 0k] 

265.91  0 .7 ) > 

1 .4 ( 0.3) 

15. 7(  1.1) 

56. 4(  1 . 2 ) > 

43. 6(  1 . 2 ) < 

1990 

2675 

0 . 4 { 0.0)  | 

[ 2k] 

260. 0(  0.6) 

1 . 6 ( 0.3) 

14. 5 { 0.7) 

48.81  1.0) 

51. 2(  1.0) 

South  Carolina 

1992 

2625 

1 . 8 ( 0.0)  1 

t 2k] 

260. 8(  1.0) 

1.7(  0.5) 

14. 9(  1.0) 

47. 8{  1.3) 

52. 2(  1.3) 

1990 

0 

0.0(  0.0) 

[ Ok] 

****♦<  0.0) 

1992 

2485 

2 . 3 ( 0.1) 

t 3k] 

258. 8(  1.4) 

1.0(  0.4) 

11. 6{  1.0) 

46. 8(  1.9) 

53.21  1.9) 

1990 

0 

0.0<  0.0) 

[ Ok] 

«****(  o.O) 

Texas 

1992 

2614 

8 . 8 ( 0.3) 

[ 3k] 

264. 6(  1 . 3)  > 

3 . 2 ( 0.6) 

18. K 1 . 2 ) > 

52. 7(  1 . 5 ) > 

47. 3(  1 . 5 ) < 

1990 

2542 

10. 2(  0.3) 

t 3k] 

258. 2(  1.4) 

1 . 6 ( 0.3) 

12. 7(  1.1) 

45. 5(  1.6) 

54.54  1.6) 

Utah 

1992 

2726 

1 . 2 ( 0.0) 

t 3k] 

274. 3(  0.7) 

2 . 3 ( 0.4) 

22. 3(  1.0) 

66. 8(  1.2) 

33.24  1.2) 

1990 

0 

0.0(  0.0) 

[ Ok] 

****♦(  0.0) 

***•*(  o.O) 

Virginia 

1992 

2710 

2 . 8 ( 0.1) 

[ 4k] 

267.91  1.2) 

2 . 7 ( 0.6) 

19.21  1.1) 

56. 7(  1.7) 

43.34  1.7) 

1990 

2661 

3 . 0 ( 0.1) 

[ 3k] 

264. 3(  1.5) 

3 . 7 ( 0.8) 

17.3 ( 1.6) 

51. 6(  1.7) 

48. 4t  1.7) 

West  Virginia 

1992 

2690 

0 . 9 { 0.0) 

t 3k] 

259. 1(  1.0)> 

0 . 6 ( 0.2) 

9 . 8 ( 0.8) 

46. 8(  1 . 6) > 

53.24  1 . 6 ) < 

1990 

2600 

l.K  0.0) 

t 3k] 

255. 9(  1.0) 

0 . 9 { 0.2) 

9 . 3 ( 0.8) 

41.81  1.1) 

58.21  1.1) 

Wisconsin 

1992 

2814 

2 . 3 ( 0.1) 

[ 6k] 

277. 9{  1.5) 

3 . 2 ( 0.6) 

27.01  1.4) 

70. 8(  2.1) 

29. 2(  2.1) 

1990 

2750 

2 . 4 ( 0.1) 

[ 3k] 

274. 5(  1.3) 

3 . K 0.4) 

23. 2(  1.4) 

65. 7(  1.6) 

34 . 3 ( 1.6) 

Wyoming 

1992 

2444 

0 . 3 ( 0.0) 

t 2k] 

275. 1(  0 . 9 ) > 

1 . 9 ( 0.4) 

21. 0(  l.D 

67. 2(  1.3) 

32.81  1.3) 

1990 

2701 

0 . 3 ( 0.0) 

t lk] 

272  If  0.7) 

1.7 ( 0.2) 

18. 5 ( 0.9) 

63. 7(  1.3) 

36.31  1.3) 

Guam 

1992 

1496 

0 . 1 ( 0.0) 

[ lk] 

235. 1{  1.0)> 

0 . 5 ( 0.1) 

5 . 6 ( 0 . 6 ) > 

25. 3(  1.4) 

74.7{  1.4) 

1990 

1617 

0 . 1 ( 0.0) 

[ Ok] 

2 j i . 8 ( 0.7) 

0 . 4 ( 0.2) 

3 . 8 ( 0.4) 

21. 7(  1.0) 

78.31  1.0) 

Virgin  Islands 

1992 

1479 

0.1{  0.0) 

[ 0k] 

222. 8(  1.1)> 

0.0(  0.1) 

0 . 6 ( 0.3) 

9 . 4 < 0.9) 

90.61  0.9) 

1990 

1326 

0 . 1 ( 0.0) 

I 0k] 

218. 7(  0.9) 

0 . 1 ( 0.1) 

0 . 7 ( 0.3) 

7.61  1.0) 

92.41  1.0) 

> INDICATES  A 

SIGNIFICANT 

INCREASE 

<OR  DECREASE  »<•) 

BETWEEN  1990  AND 

1992 

BEST  COPY  AVAILABL 


1 I $ 


Appendix  H 


THE  INFORMATION  WEIGHTING  ERROR 

Susan  C.  Loomis,  Luz  Bay,  and  Wen-Hung  Chen 
American  College  Testing 


The  Error 

In  the  process  of  recomputing  the  reading  cutscores  set  in  1992  for  the  three  achievement 
levels,  an  error  in  the  information  weighting  function  was  detected.  The  error  affected  data  for  all 
achievement  levels  set  in  1992:  reading  and  mathematics.  The  Muraki  information  weighting 
function  published  in  1993  was  used  in  the  1994  programs  to  compute  achievement  levels,  so 
only  1992  levels  are  affected. 

The  procedures  used  for  1992  were  printed  and  reported  in  numerous  places.  No  one  had 
detected  an  error.  The  psychometrician  who  developed  the  programs  for  the  1994  process  used 
Muraki’ s information  weighting  function  because  he  found  it  to  be  more  straightforward  than  the 
1 992  procedure. 

The  1992  equation1  is  as  follows: 

I;(9)  = D2ct/£. 

(*=/ 

The  1994  equation  is  as  follows: 

m, 

lj(0)  = D2cij  £/ Tr  - Tj  (6)]2Pic  (0)  , 

t = / 


where  T is  the  expected  score  for  item;  or  proficiency  9 


Tj  (9) 


£7 rPjrfO) 


1=1 


and  Tc  is  the  score  assigned  to  the  response  category  c. 


‘ Notations  for  this  equation  were  modified  to  correspond  to  those  of  Equation  2.  The  reader  will  need  to  refer  to  the 
articles  (Luecht,  1993,  and  Muraki,  1993)  for  a complete  explanation  of  the  equations. 
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Analysis  of  the  Error:  Magnitude 

The  differences  in  achievement  levels  reported  for  1992  and  the  corrected  achievement 
levels  are  due  both  to  the  error  in  item  parameters  and  to  the  error  in  information  weights.  The 
cutscores  and  percentages  of  students  scoring  at  or  above  each  for  each  achievement  level  are 
reported  in  Tables  H-l  and  H-2.  Data  in  Table  H-l  are  the  previously  reported  (incorrect)  data, 
and  data  in  Table  H-2  are  the  corrected  data. 

Table  H-l 

Mathematics  Outpoints  and  Percents  At  or  Above  as  Reported 


Grade 

Basic 

Proficient 

Advanced 

4 

Cutpoint 

211 

248 

280 

%^92  Dist 

61 

18 

2 

8 

Cutpoint 

256 

294 

331 

%>92  Dist 

63 

25 

4 

12 

Cutpoint 

287 

334 

366 

%>:92  Dist 

64 

16 

2 

Table  H-2 

Corrected  Mathematics  Outpoints  and  Percents  At  or  Above  as  Reported 

Grade 

Basic 

Proficient 

Advanced 

4 

Cutpoint 

214 

249 

282 

%>:92  Dist 

59 

18 

2 

8 

Cutpoint 

262 

299 

333 

%>92  Dist 

58 

21 

3 

12 

Cutpoint 

288 

336 

367 

%>92  Dist 

64 

15 

2 

The  corrected  cutscores  are  consistently  the  same  or  higher  than  those  previously 
reported.  The  maximum  difference  in  cutscores  originally  reported  and  the  corrected  cutscores  is 
found  for  grade  8 at  the  Basic  level: 

((original  cutscore  = 256)  - (corrected  cutscore  = 262))  = -6  points. 

The  differences  attributable  to  each  error  (parameter  estimates  and  information  weighting) 
appear  to  be  rather  small  in  most  cases. 

Table  H-3  reports  the  differences  in  cutscores  due  to  the  two  errors,  examined  one  at  a 
time.  Relative  to  the  correct  data,  the  information  weighting  error  generally  resulted  in  a lower 
composite  cutscore,  and  the  recoding  error  resulting  in  incorrect  item  parameters  generally 
resulted  in  an  even  lower  composite  cutscore. 


/ /i  if* 
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Table  H-3 

Composite  NAEP  Scale  Cutpoint  Differences  in  Mathematics  Due  to  Errors 


Achievement  Level  Cutpoint 

Information  Weighting1 

Item  Parameters2 

Grade  4 

Basic 

-2 

-5 

Proficient 

-1 

-1 

Advanced 

-1 

-1 

Grade  8 

Basic 

-4 

-8 

Proficient 

-3 

-2 

Advanced 

-1 

-2 

Grade  12 

Basic 

0 

-4 

Proficient 

-1 

-2 

Advanced 

-1 

-2 

' Difference  = Incorrect  - Correct,  based  on  correct  item  parameters.  If  the  recoding  of  data  had  been  correct,  the 
cutpoints  would  have  been  in  error  by  these  amounts,  due  to  the  incorrect  information  weighting  function. 

2 Difference  = Incorrect  - Correct,  based  on  correct  information  weights.  If  the  correct  information  weighting  function 
had  been  used,  the  cutpoints  would  have  been  in  error  by  these  amounts  due  to  the  recoding  error  resulting  in  incorrect 
item  parameters. 


Table  H-4  show  comparisons  of  percentages  of  students  who  scored  at  or  above  each 
achieveme'it  level  in  1992.  The  center  bar  on  each  of  these  graphs  shows  student  performance 
relative  to  each  achievement  level  in  1992  using  both  correct  item  parameters  and  correct 
information  weights.  The  bar  on  the  left  shows  student  performance  relative  to  achievement 
levels  computed  with  the  correct  item  parameters  and  incorrect  information  weights.  The  bar  on 
the  right  shows  student  performance  relative  to  achievement  levels  computed  with  the  incorrect 
item  parameters  and  the  correct  information  weights. 
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Table  H-4 

NAEP  Mathematics  Achievement  Levels: 
Cutpoints  and  1992  Distribution  Data 


GRADE 4 


Basic 

Outpoint 

Proficient 

Advanced 

Percent  At  or  Above  Cutpoint 
Basic  Proficient  Advanced 

Correct  Data,  Incorrect  Weight 

212 

248 

281 

62.2 

19.7 

2.1 

Correct  Data,  Correct  Weight 

214 

249 

282 

60 

18.7 

1.9 

Incorrect  Data,  Correct  Weight 

209 

248 

281 

65.6 

19.7 

2.1 

GRADE  8 


Cutpoint 

Percent  At  or  Above  Cutpoint 

Basic 

Proficient 

Advanced 

Basic 

Proficient 

Advanced 

Correct  Data,  Incorrect  Weight 

258 

296 

332 

62.6 

24.4 

3.6 

Correct  Data,  Correct  Weight 

262 

299 

333 

58.6 

21.8 

3.3 

Incorrect  Data,  Correct  Weight 

254 

297 

331 

66.5 

23.5 

3.8 

GRADE  12 

Cutpoint 

Percent  At  or  Above  Cutpoint 

Basic 

Proficient 

Advanced 

Basic 

Proficient 

Advanced 

Correct  Data,  Incorrect  Weight 

288 

335 

366 

64.8 

16.1 

1.9 

Correct  Data,  Correct  Weight 

288 

336 

367 

64.8 

15.4 

1.8 

Incorrect  Data,  Correct  Weight 

284 

334 

364 

68.7 

16.9 

2.3 

Analysis  of  the  Error:  The  Information  Weighting  Functions 

Various  analyses  were  conducted  to  determine  what,  if  any,  general  conclusions  could  be 
drawn  to  help  inform  users  of  NAEP  achievement  levels  data  about  the  factors  related  to 
differences  in  cutscores  due  to  the  information  weighting  error. 

Item  ratings  are  collected  from  two  groups  of  panelists  at  each  grade  level.  These  groups 
are  called  item  rating  groups,  and  panelists  are  assigned  to  an  item  rating  group  so  that  the  two 
are  as  equivalent  as  possible  in  terms  of  panelist  type  (teacher,  career  educator,  or  general  public; 
gender;  rac'_/ethnicity;  and  region  of  residence).  These  item  rating  groups  rate  slightly  over  half 
of  all  items  at  their  grade  level.  Item  rating  pools  are  developed  so  that  the  items  in  each  are  as 
equivalent  as  possible  in  terms  of  item  difficulty,  item  format  (multiple  choice,  short  constructed 
response,  and  extended  constructed  response),  test  time  for  the  block,  and  so  forth.  Item  blocks 
remain  in  tact  for  the  item  rating  pools.  At  least  one  block  (a  “common  block”)  is  rated  by  all 
panel  members  (i.e.,  both  item  rating  groups,  in  the  grade  group). 

Item  ratings  are  placed  on  the  NAEP  scale  by  computing  a theta  value  for  the 
dichotomous  items  and  for  the  polytomous  items  in  each  subscale  for  each  rating  group. 
Information  weights  are  applied  for  the  polytomous  items  at  the  subscale  level  before  computing 
the  subscale  score  for  both  dichotomous  and  polytomous  items. 


420 


Table  H-5  presents  the  information  weights  computed  for  each  rating  group  and  each 
subscale  for  the  mathematics  NAEP  achievement  levels.  Those  data  show  that  there  is  a 
consistent  pattern  of  error  caused  by  the  incorrect  information  function  for  math.  Analyses  of  the 
data  from  1992  and  1994  Trial  State  Assessments  in  reading  show  the  corrected  cutscores  are 
consistently  neither  higher  nor  lower  as  a result  of  this  error,  although  the  impact  of  the  error  was 
generally  to  estimate  a higher  cutscore  for  the  polytomous  items.  Simulated  item  parameters  were 
used  to  demonstrate  that  there  was  no  consistent  effect  of  the  incorrect  information  function  to 
weights  used  to  form  the  final  composite  cutpoints. 


Analysis  of  the  Effect  of  Item  Discrimination 

Figures  H-l  through  H-3  show  graphs  for  the  correct  and  the  incorrect  information 
functions  holding  other  parameters  constant  while  varying  the  item  discrimination  parameter  in 
the  generalized  partial  credit  item  response  theory  (IRT)  model.  In  general,  the  differences 
between  correct  and  incorrect  weights  increase  as  item  discrimination  increases. 

These  figures  show  no  consistent  pattern  in  the  direction  (positive  or  negative)  of  the 
difference  between  the  correct  and  incorrect  information  function  based  on  item  discrimination 
When  the  correct  information  function  is  greater  than  the  incorrect  function,  we  observe  that 
information  is  in  the  area  of  maximum  information.  It  is  not  the  case,  however,  that  the  correct 
function  always  results  in  a greater  information  weight  where  information  is  maximized. 
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Table  H-5 

Comparison  Between  the  Incorrect  and  Correct  Information  Weights  for  1992  Mathematics  Achievement  Levels 


Figure  H-l 

Comparison  Between  the  Incorrect  and  Correct  Information  Functions 
Using  Hypothetical  Item  Parameters  a=0.2,  b-0,  d0~0,  dl=2,  d2-0t  and  d3~-2,  in  Case  of  Four  Response 
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Figure  H-2 

Comparison  Between  the  Incorrect  and  Correct  Information  Functions 
Using  Hypothetical  Item  Parameters  a=0.5,  b=0,  d0=0,  dl=2,  d2=0>  and  d3=-2,  in  Case  of  Four  Response 


O 


r 


iO 

- ^ 

- CO 

- CM 


to 

o 

JO 

3 


2 

o © 
iE 


CM 

i 


00 

i 


i 


lO 

i 


i i I t— h- 

<5  lO  'sT  CO  CM 

(UO||DlUiO)U|)| 


o 

l 


-iv' 


VLV 


Figure  H-3 

Comparison  Between  the  Incorrect  and  Correct  Information  Functions 
Using  Hypothetical  Item  Parameters  a- 7,  b-Oy  d0~0,  dl~2t  d2~0,  and  d3~ 
in  Case  of  Four  Response  Categories 
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Analysis  of  the  Effect  of  Location  Parameters 

Figures  H-4  through  H-6  show  the  correct  and  incorrect  information  weighting  functions 
for  varying  location  parameters.  The  location  parameters  only  shift  the  distribution  of 
information,  and  that  is  the  case  for  both  the  correct  and  incorrect  information  weighting 
functions.  The  amount  of  difference  between  the  two  is  unchanged;  only  the  locations  change. 


Analysis  of  the  Effect  of  the  Threshold  Parameters. 

If  the  threshold  parameters  are  close,  in  terms  of  the  locations  of  ICCs,  the  correct 
information  function  will  have  a high  peak.  In  the  area  of  maximum  information,  i.e.  around  the 
peak  of  the  distribution,  the  incorrect  information  function  underestimates  information. 

As  can  be  seen  in  Figures  H-6  through  H-8,  when  the  threshold  parameters  are  relatively 
far  apart,  the  information  is  relatively  low  and  the  distribution  is  multimodal.  When  the  threshold 
parameters  are  closer  information  is  higher  and  the  distribution  tends  to  be  more  unimodal.  As 
the  threshold  parameters  move  even  closer,  the  difference  between  the  correct  and  incorrect 
information  functions  increases.  As  the  threshold  parameters  become  closer,  the  rate  at  which  the 
incorrect  weighting  function  increments  weights  at  the  peak  of  the  distribution  is  slower  than  that 
for  the  correct  function.  This  results  in  a negative  difference  between  the  two  functions  in  the 
area  of  maximum  information. 


Analysis  of  the  Error:  Conclusions 

Three  general  conclusions  can  be  drawn  from  our  analyses: 

l.  1.  No  consistent  pattern  of  over-  or  underestimation  can  be  predicted  from  the 
error  in  information  weights.  Generally,  the  incorrect  information  weighting 
function  results  in  a lower  information  weight  for  1992  mathematics 
assessment  items,  whereas,  it  results  a higher  information  weight  for  1992 
Reading  assessment  items. 

2.  The  difference  between  the  correct  and  incorrect  weights  increases  as  item 
discrimination  increases. 

3.  The  impact  of  the  incorrect  information  weighting  function  on  the  cutscores 
is  not  consistent.  The  impact  depends  upon  the  location  of  the  cutscore  and 
the  relative  weight  of  the  dichotomous  items. 


Figure  H-4 

Comparison  Between  the  Incorrect  and  Correct  Information  Functions 
Using  Hypothetical  Item  Parameters  a=1.5 , b=0,  d0~0,  dl=2,  d2=0 , and  d3~ 
in  Case  of  Four  Response  Categories 
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Fijjure  H-6 

Comparison  Between  the  Incorrect  and  Correct  Information  Functions 
? Hypothetical  Item  Parameters  a=  1.5.  b-I.  d0=0  dl=7  d?=n  d 


Incorrect 

Correct 

shaded  Difference 


in  Case  of  Four  Response  Categories 
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Figure  H-8 

Comparison  Between  the  Incorrect  and  Correct  Information  Functions 
Using  Hypothetical  Item  Parameters  a=1.5,  b=0,  d0=0,  dl=0.52,  d2=0,  and  d3=-0.52.  In  Case  of  Four 

Response  Categories 
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Appendix  I 


SAMPLE  DESIGN  AND  SELECTION  TABLES 

John  Burke  and  James  Green 
Westat , Inc. 


This  appendix  contains  the  urbanization  classifications  used  within  each  jurisdiction  and 
grade  (Tables  1-1  and  1-2).  Urbanization  classification  was  created  by  collapsing  type  of  location 
categories  as  necessary  and  according  to  specific  rules  until  each  urbanization  stratus  included  a 
minimum  of  10  percent  of  eligible  students  in  the  participating  jurisdiction. 

Also  included  in  this  appendix  is  information  on  metro  area  status  stratification  for  the 
participating  jurisdictions  (Tables  1-3  and  1-4).  All  schools  in  the  sampling  frame  were  assigned  a 
metro  area  status  based  on  their  Federal  Information  Processing  Standards  (FIPS)  county  code 
and  Office  of  Management  and  Budget  (OMB)  Metropolitan  Atea  Definitions  as  of  June  30, 
1993.  This  field  indicated  if  a school  was  located  within  a metropolitan  area  or  not. 

Tables  1-5  and  1-6  include  information  about  the  number  of  substitutes  provided  in  each 
jurisdiction.  Of  the  47  participating  jurisdictions,  42  were  provided  with  at  least  one  substitute  at 
grade  4,  and  41  were  provided  with  at  least  one  substitute  at  grade  8.  Among  jurisdictions 
receiving  no  substitutes,  the  majority  had  100  percent  participation  from  the  original  sample.  The 
number  of  substitutes  provided  to  a jurisdiction  ranged  from  0 to  24  in  the  fourth-grade  sample. 

A total  of  243  substitutes  were  selected. 


Tables  1-7  through  I- 10  show  the  number  of  schools  in  the  fourth-  and  eighth-grade 
mathematics  samples,  together  with  school  response  rates  observed  within  participating 
jurisdictions.  The  tables  also  show  the  number  of  substitutes  in  each  jurisdiction  that  were 
associated  with  a nonparticipating  original  school  selection,  and  the  number  of  those  that 
participated. 

This  appendix  also  contains  the  distribution  of  the  student  samples  and  response  rates  by 
grade,  school  type,  and  jurisdiction  in  Tables  1- 1 1 through  1-14. 
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Table  1-1 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 


Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  I-l  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 


Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued; 

Distribution  o f Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 


C' 


IT, 


c\> 

LO 


Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Fourth  Grade 
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Table  1-1  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Fourth  Grade 
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Table  1-2 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Eighth  Grade 
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Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Eighth  Grade 
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Small  Large  Urban  Fringe  of  Large/M  id- Size  Central  City  Medium 

Small  Large  Urban  Fringe  of  Large/Mid-Size  Central  City  High 

Small  Large  Largc/Small  Tovvn/Rural  Low 

Small  Large  Large/Small  Town/Rural  Medium 

Small  Large  Large/Small  Town/Rural  High 


Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Eighth  ( ule 
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Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Eighth  Grade 
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Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Eighth  Grade 
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Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Eighth  Grade 
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Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata.  Eighth  Grade 
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Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Eighth  Grade 
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Table  1-2  (continued) 

Distribution  of  Selected  Public  Schools  by  Sampling  Strata,  Eighth  Guide 
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Distribution  of  Selected  Public  Schools  by  Sampling  Strata , Eighth  Grade 
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Distribution  of  Selected  Public  Schools  by  Sampling  Strata.  Eighth  Grade 
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Table  1-3 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Fourth  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Type 

Original  Selected 
School 

Alaska 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

10 

Laige 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

i Other  Nonpublic 

Alabama 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

2 

Large 

In  Metro  Area 

Other  Nonpublic 

7 

Large 

In  Metro  Area 

Catholic 

i 

Arkansas 

Small 

In  Metro  Area 

Other  Nonpublic 

2 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

3 

Large 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

3 

Large 

In  Metro  Area 

Catholic 

2 

Arizona 

Small 

In  Metro  Area 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

Large 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

3 ' 

Large 

In  Metro  Area 

Catholic 

2 
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Table  1-3  (continued) 

Distribution  of  Selected  Nonpuhlie  Schools  by  Sampling  Stroke  Fourth  Grade 


Small  or 
Large  School 

School 

Type 

Original  Selected 
School 

California 

Small 

In  Metro  Area 

Catholic 

I 

Small 

In  Metro  Area 

Other  Nonpuhlie 

s 

Large 

In  Metro  Area 

Other  Nonpuhlie 

7 

Large 

In  Metro  Area 

Catholic 

() 

Colorado 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpuhlie 

\ 

Small 

Not  In  Metro  Area 

Other  Nonpuhlie 

Large 

Not  In  Metro  Area 

Catholic 

1 

Large 

In  Metro  Area 

Other  Nonpuhlie 

4 

Large 

In  Metro  Area 

Catholic 

4 

Connecticut 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpuhlie 

4 

Large 

Not  In  Metro  Area 

Catholic 

1 

Large 

In  Metro  Area 

Other  Nonpuhlie 

4 

Large 

In  Metro  Area 

Catholic 

0 

Delaware 

Small 

In  Metro  Area 

Other  Nonpuhlie 

12 

Small 

Not  In  Metro  Area 

Other  Nonpuhlie 

4 

Large 

In  Metro  Area 

Other  Nonpuhlie 

1 1 

Large 

In  Metro  Area 

Catholic 

17 

District  of  Columbia 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpuhlie 

s 

Large 

In  Metro  Area 

Otner  Nonpuhlie 

1 1 

Large 

In  Metro  Area 

Catholic 

1- 

Florida 

Small 

In  Metro  Area 

Other  Nonpuhlie 

s 

Small 

Not  In  Metro  Area 

Other  Nonpuhlie 

1 

Large 

In  Metro  Area 

Other  Nonpuhlie 

0 

Large 

In  Metro  Area 

Catholic 

1 

(Georgia 

Small 

In  Metro  Area 

Other  Nonpuhlie 

y 

Small 

Not  In  Metro  Area 

Other  Nonpuhlie 

Large 

Not  In  Metro  Area 

Other  Nonpuhlie 

'1 

I .arge 

In  Metro  Area 

Other  Nonpuhlie 

0 

Largo 

In  Metro  Area 

Catholic 

1 

4X1 
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Tabic  1-3  (continued) 

Disn  ibtuion  oj  Selected  NoapubUt  Schools  by  Sampling  Simla,  h'tmnh  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Original  Selected 
School 

Hawaii 

....  

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  A lea 

Other  Non  public 

Small 

Not  In  Metro  Area 

Ollier  Nonpublic 

Small 

Not  In  Metro  Area 

Catholic 

I 

Large 

Not  In  Metro  Area 

Catholic 

J 

Large 

Not  In  Metro  Area 

Other  Non  public 

i 

Large 

In  Metro  Area 

( )(her  Non  public 

n 

I .arge 

In  Metro  Area 

Catholic 

(' 

Iowa 

Small 

in  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

( )t  her  Non  public 

1 

Small 

Not  In  Metro  Area 

( )ther  N(  mi  public 

s 

S 

Small 

Not  In  Metro  Area 

Catholic 

■'i 

Large 

Not  In  Metro  Area 

Catholic 

4 

Large 

Not  In  Metro  Area 

( )ther  Nonpublic 

I an  ge 

In  Metro  Area 

Other  Nonpublic 

1 

1 aree 

In  Metro  Area 

Catholic 

s 

Indiana 

i 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

( )ther  Nonpublic 

(i 

Small 

Not  In  Metro  Area 

( Hirer  Nonpublic 

4 

I .urge 

Not  In  Metro  Area 

Catholic 

l arge 

In  Metro  Area 

( )t her  Nonpublic 

4 

1 .at  ge 

In  Metro  Area 

( 'atholic 

7 

Kentucky 

Small 

In  Metro  Area 

( atholic 

1 

Small 

In  Metro  Area 

Other  Nonpublic 

1 

Small 

Not  In  Metro  Area 

( Hirer  Nonpublic 

I arge 

Not  In  Metro  Area 

Catholic 

1 

l .aree 

Not  In  Metro  Area 

( Hher  Nonpublic 

1 .arge 

In  Metro  Area 

( Hher  Nonpublic 

* 

I .aige 

In  Metro  Area 

(.'atholic 

s 

Louisiana 

Small 

In  Moiio  Atea 

C ’atholic 

1 

Small 

In  Metro  Aiea 

( Hher  Nunpubln 

> 

Small 

Not  In  Metio  Area 

Other  Nonpublic 

1 

I arge 

Not  In  Metro  Area 

( ’atholic 

1 

1 arge 

Not  In  Metro  Atea 

( Hirer  Nonpublic 

1 

1 aige 

In  Metio  Aiea 

Other  Nonpublic 

s 

I .arec 

In  Metro  Aiea 

( atholic 

14 

Table  1-3  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Fourth  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Type 

Massachusetts 

Small 

In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Catholic 

Maryland 

Small 

In  Metro  Area 

Other  Nonpublic 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Catholic 

Maine 

Small 

In  Metro  Area 

Catholic 

Small 

In  Metro  Area 

Other  Non  public 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

Small 

Not  In  Metro  Area 

Catholic 

Large 

Not  In  Metro  Area 

Catholic 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Catholic 

Michigan 

Small 

In  Metro  Area 

Catholic 

Small 

In  Metro  Area 

Other  Nonpublic 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

Small 

Not  In  Metro  Area 

Catholic 

Large 

Not  In  Metro  Area 

Catholic 

Large 

In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Catholic 

Minnesota 

Small 

In  Metro  Area 

Catholic 

Small 

In  Metro  Area 

Other  Nonpublic 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

Small 

Not  In  Metro  Area 

Catholic 

Large 

Not  In  Metro  Area 

Catholic 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Catholic 
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Table  1-3  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata , Fourth  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Type 

Original  Selected 
School 

Missouri 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Small 

Not  In  Metro  Area 

Catholic 

2 

Large 

Not  In  Metro  Area 

Catholic 

I 

Large 

In  Metro  Area 

Other  Nonpublic 

5 

Large 

In  Metro  Area 

Catholic 

10 

Mississippi 

Smail 

In  Metro  Area 

Other  Nonpublic 

1 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

2 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Catholic 

l 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

6 

Large 

In  Metro  Area 

Other  Nonpublic 

2 

Large 

In  Metro  Area 

Catholic 

2 

Montana 

Small 

In  Metro  Area 

Other  Nonpublic 

2 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

9 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Catholic 

3 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

3 

Large 

In  Metro  Area 

Catholic 

1 

North  Carolina 

Small 

In  Metro  Area 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

2 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

6 

Large 

In  Metro  Area 

Catholic 

2 

North  Dakota 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

7 

Small 

Not  In  Metro  Area 

Catholic 

3 

Large 

Not  In  Metro  Area 

Catholic 

2 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Motto  Area 

Catholic 

3 
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Table  1-3  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Fourth  Grade 


Small  or 
Large  School 

Nebraska 

Small 

Small 

Small 

Small 

Large 

Large 

Large 

Large 

New  Jersey 
Small 
Small 
Large 
Large 

New  Mexico 
Small 
Small 
Small 
Small 
Large 
Large 
Large 
Large 

Nevada 

Small 

Small 

Large 

Large 

New  York 
Small 
Small 
Small 
Large 

Large 


Metro 

Status 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


School 
Type 

Catholic 

Other  Nonpublic 

Other  Nonpublic 

Catholic 

Catholic 

Other  Nonpublic 

Other  Nonpublic 

Catholic 


Catholic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 

Other  Nonpublic 

Other  Nonpublic 

Catholic 

Catholic 

Other  Nonpublic 

Other  Nonpublic 

Catholic 


Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 

Other  Nonpublic 
Other  Nonpublic 
Other  Nor, public 
Catholic 


Table  1-3  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata , Fourth  Grade 


Small  or 
Large  School 
Oregon 

Small 

Small 

Large 

Large 

Large 

Pennsylvania 

Small 

Small 

Small 

Small 

Large 

Large 

Large 

Rhode  Island 

Small 

Small 

Small 

Large 

Large 

Large 

South  Carolina 

Small 

Small 

Large 

Large 

Large 

Tennessee 

Small 

Small 

Small 

Large 

Large 

Large 

Texas 

Small 

Large 

Large 


Metro 

Status 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


School 

Type 

Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 
Catholic 

Other  Nonpublic 
Catholic 


Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 

Other  Nonpublic 
Catholic 


Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 

Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Other  Nonpublic 
Other  Nonpublic 
Catholic 


Original  Selected 
School 


4 

4 

1 

4 

2 


2 

8 

2 

1 

1 

5 

16 


2 

3 

1 

1 

n 

L. 

12 


4 
2 
3 

5 

1 


1 

3 

2 

1 

7 

1 


3 

5 

2 


Table  1-3  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata.  I'ourth  Grade 


Small  or 
Large  School 

Metro 

Status 

Utah 

Small 

In  Metro  Area 

Small 

Not  In  Metro  Area 

Large 

Not  In  Metro  Area 

Large 

In  Metro  Area 

Large 

In  Metro  Area 

Virginia 

Small 

In  Metro  Area 

Small 

Not  In  Metro  Area 

Large 

Not  In  Metro  Area 

Large 

In  Metro  Area 

Large 

In  Metro  Area 

Vermont 

Small 

In  Metro  Area 

Small 

In  Metro  Area 

Small 

Not  In  Metro  Aiea 

Small 

Not  In  Metro  Area 

Large 

Not  In  Metio  Area 

I .arge 

Not  In  Metio  Area 

Large 

In  Metro  Area 

1 .arge 

In  Metro  Aiea 

Washington 

Small 

In  Metio  Aiea 

Small 

Not  In  Metro  Area 

Small 

Not  In  Metro  Aiea 

I .arge 

In  Metio  .Area 

l arge 

In  Metro  Aiea 

Wisconsin 

Small 

in  Metro  Aiea 

Small 

In  Metro  Aiea 

Small 

Not  In  Metio  Area 

Small 

Not  In  Metro  Aiea 

1 aige 

Not  In  Metio  Area 

1 .arge 

Not  In  Metio  Aiea 

1 arge 

In  Metro  Atea 

1 .aree 

In  Metro  A re.  i 

School 

Tvpe 

Original  Selected 
School 

Other  Nonpublic 

A 

Other  Nonpublic 

I 

Catholic 

1 

Other  Nonpuhlie 

3 

Catholic 

"> 

Other  Non  public 

4 

Other  Nonpublic 

Other  Nonpublic 

I 

Other  Nonpuhlie 

S 

Catholic 

A 

Catholic 

1 

Other  Non  public 

i 

Other  Nonpublic 

7 

Catholic 

1 

Catholic 

( )thcr  Nonpuhlie 

A 

Other  Nonpublic 

1 

Catholic 

2 

Other  Nonpuhlie 

-'i 

Other  Nonpublu 

1 

Catholic 

1 

( )ther  Nonpuhlie 

5 

Catholic 

4 

( 'athohe 

\ 

( )lhcr  Nonpublic 

1 

Other  Nonpuhlie 

u 

Catholic 

A 

Catholic 

Other  Nonpuhlie 

1 

t )ther  Nonpuhlie 

s 

Catholic 

in 

Table  1-3  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  fourth  Grade 


Small  or  Metro  School  Original  Selected 

Large  School  Status Type School 


West  Virginia 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpublic 

i 3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

6 

Large 

Not  In  Metro  Area 

Catholic 

1 

Larne 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large  j 

In  Metro  Area 

Catholic 

2 

Wyoming 

Small 

In  Metro  Area 

Other  Nonpublic 

1 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

5 

Small 

Not  In  Metro  Area 

Catholic 

1 

Lame 

Not  In  Metro  Area 

Catholic 

n 

Large 

In  Metro  Area 

Other  Nonpublic 

1 

Lame 

In  Metro  Area 

Catholic 

1 

Table  1-4 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Eighth  Grade 


Small  or 
Large  School 

Alaska 

Small 

Small 

Large 

Large 

Large 

Alabama 

Small 

Small 

Small 

Large 

Large 

Large 

Arkansas 

Small 

Small 

Large 

Large 

Large 


Metro 

Status 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


School 
Type 

Other  Nonpublic 
Other  Nonpublic 
Catholic 

Other  Nonpublic 
Other  Nonpublic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 


Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Original  Selected 
School 


6 

19 

1 

1 

4 


6 

o 


9 

3 


4 

4 

2 

3 

i 


Arizona 

Small 

Small 

Large 

Large 

Large 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


California 

Small 

Large 

Large 

Large 


In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


Other  Nonpublic 
Catholic 
Other  Non  public 
Catholic 


Colorado 

Small 

Small 

Small 

Large 

Large 

Large 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


Catholic 

Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


6 

5 

2 

5 

3 


9 
I 

9 

10 


I 

7 

4 

1 

6 


5 


Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata , Eighth  Grade 


Small  or 
Large  School 
Connecticut 
Small 
Small 
Small 
Large 
Large 
Large 

District  of  Columbia 
Small 
Small 
Large 
Large 

Delaware 

Small 

Small 

Small 

Large 

Large 

Florida 

Small 

Small 

Large 

Large 

Georgia 

Small 

Small 

Large 

Large 

Large 

Hawaii 

Small 

Small 

Small 

Large 

Large 

Large 

Large 


Metro 

Status 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


School 
Type 

Catholic 

Other  Nonpublic 

Other  Nonpublic 

Catholic 

Other  Nonpublic 

Catholic 


Catholic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 

Other  Nonpublic 

Other  Nonpublic 

Catholic 

Other  Nonpublic 

Other  Nonpublic 

Catholic 


Original  Selected 
School 


3 

7 

1 

2 

9 

19 


7 

14 

24 


21 

8 

13 

22 


10 

1 

12 

7 


6 

4 

2 

7 


5 

5 

i} 

3 

14 

10 


Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Eighth  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Type 

Original  Selected 
School 

Iowa 

Small 

In  Metro  Area 

Catholic 

1 

2 

Small 

In  Metro  Area 

Other  Nonpublic 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

6 

Small 

Not  In  Metro  Area 

Catholic 

I 

Large 

Not  In  Metro  Area 

Catholic 

6 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

2 

Large 

In  Metro  Area 

Catholic 

8 

Indiana 

Small 

In  Metro  Area 

Catholic 

2 

Small 

In  Metro  Area 

Other  Nonpublic 

1 1 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

7 

Large 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

5 

Large 

In  Metro  Area 

Catholic 

10 

Kentucky 

Small 

In  Metro  Area 

Catholic 

7 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Aiea 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Catholic 

I 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

7 

Large 

In  Metro  Area 

Other  Nonpublic 

4 

Large 

In  Metro  Area 

Catholic 

9 

Louisiana 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

] 

Small 

Not  In  Metro  Area 

Catholic 

j 

Large 

Not  In  Metro  Area 

Catholic 

2 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

8 

Large 

In  Metro  Area 

Catholic 

20 

Massachusetts 

Small 

In  Metro  Area 

Catholic 

2 

7 

Small 

In  Metro  Area 

Other  Nonpublic 

Large 
1 -arge 

Not  In  Metro  Area 

Other  Nonpublic 

1 

8 

In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Catholic 

20 

Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Eighth  Grade 


Metro 

Status 

School 

Type 

In  Metro  Area 

Catholic 

In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Catholic 

Not  In  Metro  Area 

Other  Nonpublic 

In  Metro  Area 

Other  Nonpublic 

In  Metro  Area 

Catholic 

In  Metro  Area 

Catholic 

In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Catholic 

Not  In  Metro  Area 

Catholic 

Not  In  Metro  Area 

Other  Nonpublic 

In  Metro  Area 

Other  Nonpublic 

In  Metro  Area 

Catholic 

In  Metro  Area 

Catholic 

In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Catholic 

Not  In  Metro  Area 

Catholic 

In  Metro  Area 

Other  Nonpublic 

In  Metro  Area 

Catholic 

In  Metro  Area 

Catholic 

In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Other  Nonpublic 

Not  In  Metro  Area 

Catholic 

Not  In  Metro  Area 

Catholic 

Not  In  Metro  Area 

Other  Nonpublic 

In  Metro  Area 

Other  Nonpublic 

In  Metro  Area 

Catholic 

Small  or 
Large  School 


Maryland 

Small 

Small 

Small 

Large 

Large 

Large 

Large 

Maine 

Small 

Small 

Small 

Small 

Large 

Large 

Large 

Large 

Michigan 

Small 

Small 

Small 

Small 

Large 

Large 

Large 

Minnesota 

Small 

Small 

Small 

Small 

Large 

Large 

Large 

Large 


School 


1 

8 

1 

1 

1 

11 

15 


4 

13 

1 

2 

2 

4 

3 


2 

10 

4 


8 

10 


6 

8 

1 

2 

1 

4 

10 


Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic.  Schools  by  Sampling  Strata,  Eighth  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Type 

Original  Selected 
School 

Missouri 

Small 

In  Metro  Area 

Catholic 

3 

Small 

In  Metro  Area 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Catholic 

2 

Large 

Not  In  Metro  Area 

Catholic 

2 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

6 

Large 

In  Metro  Area 

Catholic 

17 

Mississippi 

Small 

In  Metro  Area 

Other  Nonpublic 

2 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Catholic 

I 

10 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

Large 

In  Metro  Area 

Other  Nonpublic 

3 

Large 

In  Metro  Area 

Catholic 

1 

Montana 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

16 

Small 

Not  In  Metro  Area 

Catholic 

9 

Large 

Not  In  Metro  Area 

Catholic 

3 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

4 

Large 

In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Catholic 

I 

North  Carolina 

Small 

In  Metro  Area 

Other  Nonpublic 

6 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

5 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

9 

Large 

In  Metro  Area 

Other  Nonpublic 

6 

Large 

In  Metro  Area 

Catholic 

2 

North  Dakota 

Small 

In  Metro  Area 

Other  Nonpublic 

5 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

12 

Small 

Not  In  Metro  Area 

Catholic 

I 

Large 

Not  In  Metro  Area 

Catholic 

9 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

2 

Large 

In  Metro  Area 

Other  Nonpublic 

2 

Large 

In  Metro  Area 

Catholic 

5 

r 


594 


Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Eighth  Grade 


Small  or 
Large  School 

Nebraska 

Small 

Small 

Small 

Small 

Large 

Large 

Large 

Large 

New  Hampshire 
Small 
Small 
Small 
Small 
Large 
Large 
Large 


Metro 

Status 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


School 

TyPe 

Catholic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 
Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 
Other  Nonpublic 
Other  Nonpublic 
Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 


Original  Selected 
School 


1 

4 

10 

3 
6 
2 

4 
10 


1 

6 

9 

1 

2 

2 

7 


New  Jersey 
Small 
Small 
Large 
Large 

Newf  Mexico 
Small 
Small 
Small 
Small 
Large 
Large 
Large 
Large 


In  Metro  Area 
In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


In  Metro  Area 
In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 


Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 
Catholic 

Other  Nonpublic 
Other  Nonpublic 
Catholic 


4 

6 

9 

28 


9 

7 

2 

1 

3 

3 

3 


Nevada 

Small 

Small 

Large 

Large 

Large 


In  Metro  Area 
Not  In  Metro  Area 
Not  In  Metro  Area 
In  Metro  Area 
In  Metro  Area 


Other  Nonpublic 
Other  Nonpublic 
Catholic 

Other  Nonpublic 
Catholic 


3 

2 

I 

3 

4 


Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata , Eighth  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Type 

Original  Selected 
School 

New  York 

Small 

In  Metro  Area 

Catholic 

2 

Small 

In  Metro  Area 

Other  Nonpublic 

6 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

2 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

i 

Large 

In  Metro  Area 

Other  Nonpublic 

12 

Large 

In  Metro  Area 

Catholic 

21 

Oregon 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpublic 

7 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

6 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

5 

Large 

In  Metro  Area 

Catholic 

5 

Rhode  Island 

Small 

In  Metro  Area 

Catholic 

5 

Small 

In  Metro  Area 

Other  Nonpublic 

8 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Catholic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

4 

Large 

In  Metro  Area 

Catholic 

26 

South  Carolina 

Small 

In  Metro  Area 

Other  Nonpublic 

6 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

4 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

3 

Large 

In  Metro  Area 

Other  Nonpublic 

6 

Large 

In  Metro  Area 

Catholic 

2 

Tennessee 

Small 

In  Metro  Area 

Catholic 

i 

Small 

In  Metro  Area 

Other  Nonpublic 

6 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

I 

Large 

In  Metro  Area 

Other  Nonpublic 

10 

Large 

In  Metro  Area 

Catholic 

3 

AO* 


9*0 


Or 


Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata,  Eighth  Grade 


Small  or 
Large  School 

Metro 

Status 

School 

Type 

Original  Selected 
School 

Texas 

Small 

In  Metro  Area 

Catholic 

1 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

6 

I arge 

In  Metro  Area 

Catholic 

3 

Utah 

Small 

In  Metro  Area 

Other  Nonpublic 

4 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

5 

Large 

In  Metro  Area 

Catholic 

2 

Virginia 

Small 

In  Metro  Area 

Other  Nonpublic 

7 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

4 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

7 

Large 

In  Metro  Area 

Catholic 

4 

Vermont 

Small 

In  Metro  Area 

Catholic 

i 

Small 

In  Metro  Area 

Other  Nonpublic 

3 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

19 

Small 

Not  In  Metro  Area 

Catholic 

2 

Large 

Not  In  Metro  Area 

Catholic 

o 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

5 

Large 

In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Catholic 

3 

Washington 

Small 

In  Metro  Area 

Catholic 

i 

Small 

In  Metro  Area 

Other  Nonpublic 

7 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

3 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

i 

Large 

In  Metro  Area 

Other  Nonpublic 

7 

Large 

In  Metro  Area 

Catholic 

6 

a e\/ 


Table  1-4  (continued) 

Distribution  of  Selected  Nonpublic  Schools  by  Sampling  Strata.  Eighth  Grade 


Small  or 
Large  School 

Metro 

Status 

■BSHI 

Original  Selected 
School 

Wisconsin 

Small 

In  Metro  Area 

Catholic 

6 

Small 

In  Metro  Area 

Other  Nonpublic 

14 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 1 

Small 

Not  In  Metro  Area 

Catholic 

5 

Large 

Not  In  Metro  Area 

Catholic 

4 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

3 

Large 

In  Metro  Area 

Other  Nonpublic 

S 

Large 

In  Metro  Area 

Catholic 

17 

West  Virginia 

Small 

In  Metro  Area 

Catholic 

Small 

In  Metro  Area 

Other  Nonpublic 

5 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 1 

Small 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Catholic 

1 

Large 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Other  Nonpublic 

1 

Large 

In  Metro  Area 

Catholic 

3 

Wyoming 

Small 

In  Metro  Area 

Other  Nonpublic 

7 

Small 

Not  In  Metro  Area 

Other  Nonpublic 

1 1 

Small 

Not  In  Metro  Area 

Catholic 

1 

Not  In  Metro  Area 

Catholic 

1 

Not  In  Metro  Area 

Other  Nonpublic 

1 

Hi 

In  Metro  Area 

Catholic 

407 


uest  copy  available 


Table  1-5  (continued) 

Number  of  Substitute  Schools  Provided  in  Each  Jurisdiction  By  School  Type,  Fourth  Grade 
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Table  1-6 

Number  of  Substitute  Schools  Provided  in  Each  Jurisdiction  By  School  Type,  Eighth  Grade 


'JD 


Table  1-6  (continued) 

Number  of  Substitute  Schools  Provided  in  Each  Jurisdiction  By  School  Type , Eighth  Grade 
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Distribution  of  Grade  4 Nonpublic-School  Sample  by  Jurisdiction 
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Distribution  of  Grade  8 Public-School  Sample  by  Jurisdiction 
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Table  1-11 

Distribution  of  the  Fourth-Grade  Public-School  Student  Sample  and  Response  Rates  by  Jurisdiction 
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Massachusetts  95%  2,753  128  2,611  2,497 

Michigan  94%  2,653  88  2,544  2,382 

Minnesota  94%  2,686  85  2,559  2,425 

Mississippi  96%  2,927  76  2,836  2,716 

Missouri  95%  2,857  83  2,777  2,643 
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Distribution  of  the  Fourth-Grade  Public-School  Student  Sample  and  Response  Rates  by  Jurisdiction 
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Distribution  of  the  Fourth-Grade  Nonpublic-School  Student  Sample  and  Response  Rates  by  Jurisdiction 
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Distribution  of  the  Eighth-Grade  Public-School  Student  Sample  and  Response  Rates  by  Jurisdiction 
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Distribution  of  the  Eighth-Grade  Nonpublic-School  Student  Sample  and  Response  Rates  by  Jurisdiction 
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